ERC-3668 edge case for clientside http/ssl errors

[raffy]

I agree, and brought up same point with ricmoo.

But if we update the client to fix 4XX and invalid 2XX response, we need to ensure a JSON data-url is a valid endpoint, since OffchainTryNext() and OffchainLookupUnanswered() can be emulated using data-url and using recursive CCIP-Read w/pairs of endpoints:

endpoints = [a, b, c]

revert OffchainLookup w/ [a, data:0x]
   if response == 0x or invalid, revert OffchainLookup w/ [b, data:0x]
       if response == 0x or invalid, revert OffchainLookup w/[c, data: 0x]
           if response == 0x or invalid, <No Answer>

My thinking was just that if we update the client, might as well make the spec better.

[0xc0de4c0ffee]

I mean it’s old “locked” ERC, if we can update more than basic fix I’d like to push for direct ipfs://.., ipns://.., bzz://.. support as it’s limited to http/s only.

@raffy

we need to ensure a JSON data-url is a valid endpoint, since OffchainTryNext() and OffchainLookupUnanswered() can be emulated using data-url and using recursive CCIP-Read w/pairs of endpoints:

it’s already possible to do that with current specs, we shouldn’t add all possible iteration in specs even if it’s possible to update old ERCs. It’s up to devs how they want to use that callback as fallback.
gateways = [ https://gateway1/…, datauri:hex(gateway1 failed)]
callback >> retry >> fallback to gateway 2.

[nick.eth]

@raffy

a new UniversalResolver (UR.sol ) which is clean, has support for resolve(multicall), allows for arbitrary callback recursion, and provides extra information.

It’s difficult to evaluate an entirely new implementation against the current one. Can you describe how it’s different, and why the existing one can’t have PRs instead? Tagging @taytems .

@raffy

1. a proof-of-concept of a better ERC-3668 algorithm, implemented as an ethers ContractRunner (CCIPReadRunner.ts )

No general objections to this as a new EIP extending 3668. Forwards-compatibiltiy seems like an issue here: contracts may avoid using the new functionality because it will break legacy 3668-only clients, and instead opt for degraded support that they know won’t break things. Perhaps some method could be incorporated that lets the client signal to the contract that it supports this new EIP?

@raffy

• error OffchainLookupUnanswered() supplied to the callback after reaching the last endpoint without success, cannot be called from an endpoint response

How do you send an error to a contract?

@0xc0de4c0ffee

2nd huge issue, current records are signed by gateways, it should be signed by owner or using owner’s permits.

How responses are verified is entirely up to implementations; that’s not specified by 3668.

@raffy

• a 4XX error with valid {data} would trigger the callback — currently this case blows up.

How would the callback know it was being called for a 4xx? Intuitively this seems bad.

[raffy]

@nick.eth

It’s difficult to evaluate an entirely new implementation against the current one. Can you describe how it’s different, and why the existing one can’t have PRs instead?

I only wrote it as a testbed for CCIP-Read, since its a complex example of wrapping and potentially recursive calls, but it turned out as a pretty nice implementation.

@nick.eth

Perhaps some method could be incorporated that lets the client signal to the contract that it supports this new EIP?

Initially I was thinking first URL, but after brainstorming with Premm a bit, the last URL could be a signal value (indicating the new feature set) and that would be maximally backwards compatible without requiring dataurl support.

@nick.eth

How do you send an error to a contract?

Since callback(bytes response, bytes extraData) expects calldata, response can be supplied as just bytes4(selector) to indicate an error.

A successful call to f() returns (bytes4) via CCIP-Read would return 0xXXXXXXXX0000.....0000

@nick.eth

How would the callback know it was being called for a 4xx? Intuitively this seems bad.

I think the question is the difference between a CCIP-Read server that emits a 4XX and the contract is never told about the problem, only the client. And one that will pass {data: "0x..."} to the callback regardless of status code. For example, a 404 could supply {data: "0xc5723b51"} which is error NotFound().

I don’t think this feature is necessary, it just seems more useful, since the new protocol would effectively silence the error, as it will just try the next endpoint rather than killing the session.

For new stuff: since the contract decides if it accepts the request or wants another response, the only issue I can think of would be some kind of HTTP middleware throwing an error between the server and the client, and responding with 4XX with malicious {data: "0x..."} that the contract somehow accepts.

For backwards compat, nearly every contract expects the response to ABI encoded, so bytes.length % 32 > 0 will likely revert.

[taytems]

The general idea here seems pretty useful in allowing the contract to be aware of and control the response in the case of an error. I’ve run into the issue before (specifically with the UniversalResolver sender).

I’d note that the proposed solution would make it harder to debug a faulty endpoint since an HTTP error no longer becomes the source of the error to throw, but is instead given to the contract’s resulted handling of OffchainLookupUnanswered(). That isn’t crucial though and could be fixed with better logging/etc.

@raffy

1. a new UniversalResolver (UR.sol ) which is clean, has support for resolve(multicall), allows for arbitrary callback recursion, and provides extra information.

I don’t really see how this is an improvement over the pending UniversalResolver v3 changes. The implementation is more simple yes, but it also provides a significantly smaller feature set without any of the existing edge case handling.

[0xc0de4c0ffee]

This is clearly bug in ERC3668 specs…

@0xc0de4c0ffee

I’m not sure how far can we go without breaking ERC3668 specs. it’s cleverly designed to fit in between web2 & web3, so whole lookup process is trying to mimic http but we want more failsafe scenario for web3/decentralization…

Possible direct bug fix this in erc3668.

@0xc0de4c0ffee

(8) & (9) should be merged, & 4xx should use same fallback logic as 5xx.
possibly replace (8) with "if response json doesn’t contain {"data": ...}" or data value is not valid 0x prefixed hex, go back to step (5) and pick a different URL, or stop if there are no further URLs to try.

We’re working on new erc7700 draft to wrap all types of crosschain & dweb storage providers, @NameSys will push that new version soon.
Basic outline is to wrap everything with single new revert selector, eg.(bytes4(erc7700.selector)+bytes4(erc3668. selector)+…data). Adding proper interface checks for read, write & callback usages, auto retry next gateway after 4xx & 5xx and support ipfs:// bzz:// ipns:// types in gateway *if 3668 revert data is wrapped in 7700 selector.

[nick.eth]

@raffy

Since callback(bytes response, bytes extraData) expects calldata, response can be supplied as just bytes4(selector) to indicate an error.

A successful call to f() returns (bytes4) via CCIP-Read would return 0xXXXXXXXX0000.....0000

I don’t think this is true? The callback expects the response from the CCIP-read function, which will not usually be calldata.

Sending errors to contracts seems like a very odd change in the usual flow, and not something I would expect to happen, too.

@raffy

I think the question is the difference between a CCIP-Read server that emits a 4XX and the contract is never told about the problem, only the client. And one that will pass {data: "0x..."} to the callback regardless of status code. For example, a 404 could supply {data: "0xc5723b51"} which is error NotFound().

I think this amounts to ignoring the status code of the response as long as it contains a valid body? I think that’s a bad idea.

[raffy]

@nick.eth

I don’t think this is true? The callback expects the response from the CCIP-read function, which will not usually be calldata.

Okay, not calldata, but an abi-encoded response, which should be word-aligned. My repo implements this solution and I don’t see any issue disambiguating a response from an error. It’s the same as a returning an error over multicall as we’ve discussed previously.

My design only needs to encode 1 signal value: OffchainLookupUnanswered(). If I can’t assume this is word-aligned, there are other ways of communicating this information to the contract (see below).

@nick.eth

I think this amounts to ignoring the status code of the response as long as it contains a valid body? I think that’s a bad idea.

It’s perfectly fine if it doesn’t do this, but that just means that either the status 2XX or the response is ignored. Servers would instead indicate an error to the contract by responding 200. These are equivalent to me.

---

I just think that the contract should dictate which response is accepted and ultimately decide what to do if no response is sufficient.

Recall, with this change, the contract’s callback:

1. can revert OffchainTryNext(sender) to reject a response w/o terminating the OffchainLookup session.
2. may be supplied bytes that are not word-aligned (this can already happen.)
3. will be supplied OffchainLookupUnanswered() if no response is accepted

---

Taking your feedback, 2 and 3 could be replaced with: if the hash of the response bytes are exactly 0x... then the CCIP-Read client is indicating OffchainLookupUnanswered(). Ideally, that response would be less than 32 bytes, causing any abi.decode to blow up, and therefore be backwards-compatible (since any existing CCIP-Read session that got no response is currently fatal.)

[nick.eth]

@raffy

Okay, not calldata, but an abi-encoded response, which should be word-aligned. My repo implements this solution and I don’t see any issue disambiguating a response from an error. It’s the same as a returning an error over multicall as we’ve discussed previously.

My design only needs to encode 1 signal value: OffchainLookupUnanswered(). If I can’t assume this is word-aligned, there are other ways of communicating this information to the contract (see below).

Okay. Still not a fan, though I see your point. It certainly shouldn’t do this unless it can detect that the contract definitely implements this new standard, though.

@raffy

It’s perfectly fine if it doesn’t do this, but that just means that either the status 2XX or the response is ignored. Servers would instead indicate an error to the contract by responding 200. These are equivalent to me.

I’m not really sure what you mean by this.

Why not have 400s and 500s return an error in the same way you document above, with the response embedded?

[raffy]

@nick.eth

Why not have 400s and 500s return an error in the same way you document above, with the response embedded?

This also seems fine but I’m not sure what the contract would do with a human-readable error.

For an error, I see 3 possible responses:

• 400 “bad request”`
• 400: {"data": "0x..."} — my suggestion: just ignore status code and propagate whatever it sent, let the contract decide
• 200: {"data": "0x..."} — your suggestion (I thought): standard abi-encoded response that encodes there was an error, where the contract can decode it, but the HTTP layer isn’t aware of it

My thinking was, unless the server returns {"data": "0x...."} the contract can’t really digest it. It could wrap it in Error(string) but what’s the contract going to do with it?

Currently, if it’s 4XX, the 3668 session just terminates. Instead of silently ignoring the error, it seemed useful (if it was properly formatted) to relay that information to the contract.

I’m not against an HTTP status code wrapper, if that’s what you’re suggesting, it just seems like if the server is functioning, it should be relaying errors via {"data": "0x..."} rather than status code. If the server isn’t functioning, likely the error (and body) aren’t usable by the contract.

For example, with a trusted gateway, if the error isn’t signed, why would you trust it? And for an untrusted gateway, why trust anything (including the status code) about the response?

[nick.eth]

@raffy

400: {"data": "0x..."} — my suggestion: just ignore status code and propagate whatever it sent, let the contract decide

I am not a fan of any solution that involves ignoring the status code, or sending 200s when it’s actually an error.

What I was suggesting is that if you want to be able to handle errors in the contract, you can pass them in as error objects containing the data decoded from the response (if data could be decoded from the response).

@raffy

For example, with a trusted gateway, if the error isn’t signed, why would you trust it? And for an untrusted gateway, why trust anything (including the status code) about the response?

What’s the alternative to trusting the error? The process can’t proceed further without a valid response.

[0xc0de4c0ffee]

@nick.eth

What’s the alternative to trusting the error? The process can’t proceed further without a valid response.

while (req.status != 200) {
    retryNext()
}

Both 4XX & 5XX errors should auto fallback to next gateway in list.
with current specs if first gateways is throwing 4xx, it’ll stop whole lookup process.
this is more important as we’re selecting random gateways / shuffling gateway list.

& we also need that if we want to add data:application/json;charset=utf-8,%7B%22data%22%3A%220xfffffffff%22%7D data uris as 2nd gateway for callback to indicate that 1st gateway is throwing 4xx/5xx.

[raffy]

@nick.eth

What’s the alternative to trusting the error? The process can’t proceed further without a valid response.

The feature I want is to allow untrusted/unreliable gateways in the gateway set.

To allow that, the arbiter of truth has to be the contract, since a malicious gateway can just fake its response and status code.

This implies that gateway errors at the HTTP level are meaningless to the protocol, so I concluded the status code can just be ignored, and all that matters is obtaining a correctly encoded response that the contract accepts, otherwise the call is a failure.

I say “ignore” because it seems like nice developer UX if gateway errors also set their status code:

@raffy

It’s perfectly fine if it doesn’t do this, but that just means that either the status 2XX or the response is ignored. Servers would instead indicate an error to the contract by responding 200. These are equivalent to me.

If a signing key is involved, errors should be signed. Any HTTP failure simply moves on to the next endpoint.

For a proving gateway, the appropriate proofs need supplied even in the failure case, otherwise, why trust it? Linea’s gateway suffers from this problem: their gateway can claim any storage value doesn’t exist and their verifier accepts it without proof.

[nick.eth]

@raffy

The feature I want is to allow untrusted/unreliable gateways in the gateway set.

To allow that, the arbiter of truth has to be the contract, since a malicious gateway can just fake its response and status code.

This implies that gateway errors at the HTTP level are meaningless to the protocol, so I concluded the status code can just be ignored, and all that matters is obtaining a correctly encoded response that the contract accepts, otherwise the call is a failure.

Ignoring HTTP status codes is generally a bad practice, though. I’m not sure why you object to just passing them through - along with response data - to the contract and letting it decide?

@raffy

For a proving gateway, the appropriate proofs need supplied even in the failure case, otherwise, why trust it? Linea’s gateway suffers from this problem : their gateway can claim any storage value doesn’t exist and their verifier accepts it without proof.

Right, but you still have to handle the case where an error response is returned without valid proof data.

[raffy]

@nick.eth

Right, but you still have to handle the case where an error response is returned without valid proof data.

That’s ultimately handled by OffchainLookupUnanswered() — no gateway provided response that the contract accepted.

(Note: this is equivalent to having the last endpoint being a data URL where the payload is some unique value the contract can detect.)

Individual errors for each gateways would come in two types:

1. Server errors: eg. “unknown selector”, “unknown name”, “divide by zero”, etc. which could be signed or proven up to that point
   • This only works if it’s properly encoded: {"data": "0x..."}
2. HTTP errors: eg. “no response”, “server unavailable”

Neither of these should kill the gateway iteration unless the contract says so.

HTTP errors certainly could be wrapped with HTTPError(uint16 code, string message) or w/e but I thought you were against passing error data to the contract.

@nick.eth

Sending errors to contracts seems like a very odd change in the usual flow, and not something I would expect to happen, too.

@raffy

My design only needs to encode 1 signal value: OffchainLookupUnanswered(). If I can’t assume this is word-aligned, there are other ways of communicating this information to the contract (see below).

I claim the common implementation would be the following since there’s not much you can do with that error:

if bytes4(response) == HTTPError.selector) { 
    revert OffchainTryNext(address(this));
}

Unaware contracts (except for pass-through) would fail to decode that response, revert, and terminate the process, which is the current behavior.

.

[nick.eth]

@raffy

HTTP errors certainly could be wrapped with HTTPError(uint16 code, string message) or w/e but I thought you were against passing error data to the contract.

I questioned the approach, but you provided justification. I’m okay with it as long as there’s some detection mechanism so that 3668-only contracts won’t be sent error data they may not understand.