One Routing ISM, Four Chains: Crashing Consensus Nodes Through Hyperlane

A route in Hyperlane's Cosmos module could point back to itself. Verifying a message through that route called the same verifier again, with no depth bound. A gas charge was meant to stop the recursion, but an unauthenticated dry run query let the caller raise its gas limit high enough for the Go runtime to hit fatal stack overflow first. The process exited, including when that process was a consensus node.

I reproduced the crash in a local Hyperlane test chain and in four downstream protocols using this implementation: Celestia, Dymension, KYVE, and Noble. Their exposure differed. KYVE already had a route back to itself on mainnet when I checked. My Celestia and Dymension PoCs created one on local chains. Noble's permissioned transaction gate prevented an ordinary account from creating one. Once the route existed and the query service was reachable, anyone could trigger the crash remotely.

Celestia's team rated the impact High, describing it as the ability to halt any consensus node. Its v10.5.0 release shipped a patched Hyperlane fork, v1.3.1. I demonstrated that a single query could crash a consensus node, and sending it to enough reachable validators could halt the network.

A routing loop the module accepted

Hyperlane uses an Interchain Security Module (ISM) to verify messages. A Routing ISM chooses another ISM based on the message's origin domain. That is useful when different domains need different verification policies. In the route update path, the module checked that the destination ISM existed and then stored the route. It did not reject a destination equal to the Routing ISM's own ID.

183
// check if the ism we want to route to exists
184
exists, err := k.coreKeeper.IsmExists(ctx, req.Route.Ism)
185
if err != nil || !exists {
186
    return errors.Wrapf(types.ErrUnkownIsmId, "ISM %s not found", req.Route.Ism.String())
187
}
188

189
// we don't check if the domain was overwritten
190
routingISM.SetDomain(req.Route)

With domain 1 pointed back at the same ISM, a message from domain 1 followed this path:

1
Verify(routing ISM)
2
  → RoutingISMHandler.Verify(routing ISM)
3
  → Verify(routing ISM)
4
  → RoutingISMHandler.Verify(routing ISM)
5
  → ...

The routing handler selects the route, then calls the core verifier again:

32
// get the ism for the registered route
33
routedIsm, exists := routingIsm.GetIsm(message.Origin)
34
if !exists || routedIsm == nil {
35
    return false, errors.Wrapf(types.ErrNoRouteFound, "no route found for domain %d", message.Origin)
36
}
37

38
// call the top level Verify method on the core module
39
// this method will then recursively invoke the Verify method on all the sub ISMs
40
return m.keeper.coreKeeper.Verify(ctx, *routedIsm, metadata, message)

The core verifier charges 10,000 gas before dispatching to that ISM:

114
func (k Keeper) Verify(ctx context.Context, ismId util.HexAddress, metadata []byte, message util.HyperlaneMessage) (bool, error) {
115
    // Consume a fixed amount of gas to prevent DoS attacks and limit recursive calls.
116
    // This ensures a maximum number of approximately 2100 recursive verify calls.
117
    sdk.UnwrapSDKContext(ctx).GasMeter().ConsumeGas(10000, "ism verification")
118

119
    handler, err := k.ismRouter.GetModule(ismId)
120
    if err != nil {
121
        return false, err
122
    }
123
    return (*handler).Verify(ctx, ismId, metadata, message)
124
}

At an ordinary limit, the charge eventually causes a recoverable failure when gas runs out. The route was safe only if the gas meter stopped recursion before the stack filled.

The gas meter could lose the race

The VerifyDryRun query defaults to 200,000 gas, but parsed a decimal gas_limit chosen by the caller as an unsigned integer with 32 bits. Its maximum was 4,294,967,295. The code even said the meter was there to avoid stack overflows from ISMs calling themselves:

89
func (qs queryServer) VerifyDryRun(ctx context.Context, req *types.QueryVerifyDryRunRequest) (*types.QueryVerifyDryRunResponse, error) {
90
limit := uint64(200000)
91
if req.GasLimit != "" {
92
    parsed, err := strconv.ParseUint(req.GasLimit, 10, 32)
93
    if err != nil {
94
        return nil, err
95
    }
96

97
    limit = parsed
98
}
99

100
// explicitly set a GasMeter to not run into stack overflows as ISM might call themselves
101
sdkCtx := sdk.UnwrapSDKContext(ctx).WithGasMeter(storetypes.NewGasMeter(limit))

When I tested this, the limits of 1.0 and 1.1 billion exhausted gas safely. 1.2 billion crashed with fatal error: stack overflow. That threshold is an observation from these builds and this environment, not a universal Go constant. The PoC used the maximum accepted limit to make the outcome reliable. Go's stack overflow is a fatal runtime error, so it terminates the node process. The existing test of a route back to itself used a limit of 1 billion gas and therefore exercised the safe side of the boundary.

One upstream bug, four downstream applications

The shared code entered each chain through either the upstream module or a fork maintained for that chain:

Celestia: the crash on a running node

For Celestia, I started a local v10.4.0 chain with one validator and enabled the REST and gRPC query services. A funded test account created a Routing ISM and set its destination for domain 1 to its own ID. The trigger then made one unauthenticated request to /hyperlane/v1/verify_dry_run with that ISM, a message from domain 1, and the maximum gas limit. The HTTP connection dropped, and the node terminal ended in fatal error: stack overflow. The stack trace repeated the routing handler and core verifier frames.

This distinction matters operationally, the route is persistent chain state, while the final trigger is an unauthenticated query. Once a vulnerable route is present, restarting a node alone does not remove the precondition. The same query can be sent again to an exposed node after it restarts.

The patch and the remaining boundary

Celestia's fix addressed both sides of the problem. Keeper.Verify now walks Routing ISMs iteratively rather than growing the Go call stack, and VerifyDryRun rejects gas limits above 10 million. The patch preserves gas accounting for existing routing behavior, including cycles, and landed in the Hyperlane fork release v1.3.1. Celestia then bumped that dependency in celestia-app v10.5.0.

The patched Keeper.Verify follows routes in a loop. It records visited ISMs so a cycle can exhaust its gas budget without repeatedly calling Verify on the Go stack:

123
func (k Keeper) Verify(ctx context.Context, ismId util.HexAddress, metadata []byte, message util.HyperlaneMessage) (bool, error) {
124
    sdkCtx := sdk.UnwrapSDKContext(ctx)
125
    recorder := &gasRecorder{GasMeter: sdkCtx.GasMeter()}
126
    routeCtx := sdkCtx.WithGasMeter(recorder)
127

128
    // visited maps each ISM on the walk to the index in recorder.charges where its hop starts.
129
    visited := make(map[util.HexAddress]int)
130
    for {
131
        if start, ok := visited[ismId]; ok && !recorder.refunded {
132
            return verifyRoutingCycle(sdkCtx.GasMeter(), ismId, recorder.charges[start:])
133
        }
134
        visited[ismId] = len(recorder.charges)
135

136
        // Consume a fixed amount of gas to prevent DoS attacks and limit routing hops.
137
        recorder.ConsumeGas(10000, "ism verification")
138

139
        handler, err := k.ismRouter.GetModule(ismId)
140
        if err != nil {
141
            return false, err
142
        }
143
        routing, ok := (*handler).(util.RoutingInterchainSecurityModule)
144
        if !ok {
145
            return (*handler).Verify(ctx, ismId, metadata, message)
146
        }
147
        ismId, err = routing.Route(routeCtx, ismId, message)
148
        if err != nil {
149
            return false, err
150
        }
151
    }
152
}

The patched VerifyDryRun also caps the caller's requested gas before constructing its meter:

89
// MaxVerifyDryRunGasLimit bounds the gas, and therefore the work, of a single
90
// VerifyDryRun query.
91
const MaxVerifyDryRunGasLimit = 10_000_000
92

93
func (qs queryServer) VerifyDryRun(ctx context.Context, req *types.QueryVerifyDryRunRequest) (*types.QueryVerifyDryRunResponse, error) {
94
    limit := uint64(200000)
95
    if req.GasLimit != "" {
96
        parsed, err := strconv.ParseUint(req.GasLimit, 10, 32)
97
        if err != nil {
98
            return nil, err
99
        }
100

101
        if parsed > MaxVerifyDryRunGasLimit {
102
            return nil, fmt.Errorf("gas limit %d exceeds maximum %d", parsed, MaxVerifyDryRunGasLimit)
103
        }
104
        limit = parsed
105
    }
106

107
    // explicitly set a GasMeter to not run into stack overflows as ISM might call themselves
108
    sdkCtx := sdk.UnwrapSDKContext(ctx).WithGasMeter(storetypes.NewGasMeter(limit))