Skip to content

fix(grpc-web): set ReadTimeout, WriteTimeout, IdleTimeout, and MaxOpenConnections#3605

Open
amir-deris wants to merge 11 commits into
mainfrom
amir/sei-10199-fix-grpc-timeout
Open

fix(grpc-web): set ReadTimeout, WriteTimeout, IdleTimeout, and MaxOpenConnections#3605
amir-deris wants to merge 11 commits into
mainfrom
amir/sei-10199-fix-grpc-timeout

Conversation

@amir-deris

@amir-deris amir-deris commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

PR #3558 added ReadHeaderTimeout to the gRPC-Web http.Server, protecting against slow-header attacks. The body-stall DoS vector remained: a client that sends valid headers and then stalls while the server waits indefinitely for the request body. This PR closes that gap and adds a connection cap to bound total resource usage.

Timeout changes

Field Value Purpose
ReadHeaderTimeout 10s already present — slow header protection
ReadTimeout 30s primary fix — drops connections that stall after headers
WriteTimeout 2m backstop against slow-read attacks on the response side
IdleTimeout 30s reclaims idle keep-alive connections

WriteTimeout is set to 2 minutes rather than a shorter value because the longest legitimate call through this server is BroadcastTxCommit, which is bounded by Tendermint's TimeoutBroadcastTxCommit (default 10s). Paginated query RPCs are already capped at the application layer (PLT-361). The 2-minute window gives a large margin above any real workload while still providing a backstop against response-side connection holds.

MaxOpenConnections

Wraps the gRPC-Web listener with netutil.LimitListener to cap the number of simultaneously open TCP connections. New connections block at the OS accept queue once the limit is reached rather than being accepted and left waiting inside the server.

  • Default: 1000 (matches the existing default for the API and gRPC servers)
  • Operator-configurable via grpc-web.max-open-connections in app.toml; set to 0 to disable the cap

Files changed: sei-cosmos/server/grpc/grpc_web.go, sei-cosmos/server/config/config.go

Tests

TestStartGRPCWebTimeouts — starts a real gRPC-Web server on a free loopback port and asserts all four timeout fields on the returned *http.Server.

Test plan

  • go test ./sei-cosmos/server/grpc/... -run TestStartGRPCWebTimeouts -v passes
  • gofmt -s -l . prints nothing
  • make build succeeds

🤖 Generated with Claude Code

@amir-deris amir-deris self-assigned this Jun 17, 2026
@amir-deris amir-deris changed the title Added read, write and idle timeout params to grpc web server fix(grpc-web): set ReadTimeout, WriteTimeout, and IdleTimeout to prevent body-stall DoS Jun 17, 2026
@cursor

cursor Bot commented Jun 17, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes public node networking behavior (timeouts and connection limits) on a widely exposed port; mis-tuning could drop slow legitimate clients, though defaults align with existing API limits.

Overview
Hardens the gRPC-Web HTTP server against slow-body and connection-exhaustion DoS by adding ReadTimeout (30s), WriteTimeout (2m), and IdleTimeout (30s) alongside the existing ReadHeaderTimeout, and by capping simultaneous TCP connections with netutil.LimitListener.

Configuration adds grpc-web.max-open-connections (default 1000, same as the API server; 0 = unlimited) on GRPCWebConfig, wired through defaults, Viper/GetConfig, and the auto-managed app.toml template.

Startup now binds an explicit TCP listener and serves with Serve(listener) instead of ListenAndServe, applying the connection limit when configured. TestStartGRPCWebTimeouts asserts all four timeout fields on a live server instance.

Reviewed by Cursor Bugbot for commit 19c77a3. Bugbot is set up for automated code reviews on this repo. Configure here.

@amir-deris amir-deris changed the title fix(grpc-web): set ReadTimeout, WriteTimeout, and IdleTimeout to prevent body-stall DoS fix(grpc-web): set ReadTimeout, WriteTimeout, and IdleTimeout Jun 17, 2026
@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJun 18, 2026, 8:30 PM

Comment thread sei-cosmos/server/grpc/grpc_web.go Outdated
@codecov

codecov Bot commented Jun 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.27273% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.14%. Comparing base (51ce0c0) to head (19c77a3).

Files with missing lines Patch % Lines
sei-cosmos/server/grpc/grpc_web.go 66.66% 3 Missing and 2 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3605      +/-   ##
==========================================
- Coverage   59.02%   58.14%   -0.88%     
==========================================
  Files        2215     2141      -74     
  Lines      182521   173986    -8535     
==========================================
- Hits       107731   101165    -6566     
+ Misses      65094    63823    -1271     
+ Partials     9696     8998     -698     
Flag Coverage Δ
sei-chain-pr 85.49% <77.27%> (?)
sei-db 70.41% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-cosmos/server/config/config.go 94.59% <100.00%> (+0.05%) ⬆️
sei-cosmos/server/config/toml.go 57.14% <ø> (ø)
sei-cosmos/server/grpc/grpc_web.go 64.51% <66.66%> (-2.16%) ⬇️

... and 74 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread sei-cosmos/server/grpc/grpc_web.go Outdated
Comment on lines +31 to +33
ReadTimeout: 30 * time.Second,
WriteTimeout: 2 * time.Minute,
IdleTimeout: 120 * time.Second,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idle timeout strikes me as a bit long. What do you think @sei-will?

Also, do we have any data on how long standard requests take? Ideally we keep these tightly scoped to what we expect standard workloads to use. For instance, if the largest range query only take 10 seconds based on x years historical data, then we can tighten this to 15 seconds for example.

We will never get this exactly right so not looking for a perfect magic number, just curious if we have those inputs. My hunch is 2 minutes for a write is more lenient than we need to be

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for feedback. Reduced the idle timeout to 30s.

Comment thread sei-cosmos/server/grpc/grpc_web.go
@amir-deris amir-deris changed the title fix(grpc-web): set ReadTimeout, WriteTimeout, and IdleTimeout fix(grpc-web): set ReadTimeout, WriteTimeout, IdleTimeout, and MaxOpenConnections Jun 17, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9a4484d. Configure here.

Comment thread sei-cosmos/server/config/config.go
Comment thread sei-cosmos/server/grpc/grpc_web.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants