Skip to content

Use json_each to make SQLite single inserts batch inserts#1276

Open
brandur wants to merge 1 commit into
masterfrom
brandur-sqlite-batch-inserts
Open

Use json_each to make SQLite single inserts batch inserts#1276
brandur wants to merge 1 commit into
masterfrom
brandur-sqlite-batch-inserts

Conversation

@brandur
Copy link
Copy Markdown
Contributor

@brandur brandur commented Jun 5, 2026

Back when I was first implementing SQLite, the sqlc support was so
catastrophically bad that many, many basic utilities just could not be
used in queries. I tried like a half dozen different ways, but couldn't
find anything that was both supported by SQLite and sqlc, so I ended up
making a number of batch inserts into single inserts that get looped
over in the driver. This isn't as bad in SQLite because usually your
inserting to a local database (i.e. no network connections), but it's
still not ideal in that it's probably somewhat slower, and deviates from
the Postgres implementation.

Sqlc's made some SQLite improvements since then, and I found that I'm
now able to get a batch insert working json_each and json_extract.

Here, go through the existing SQLite driver function and update things
to use batch inserts wherever possible. Fixed operations:

  • JobInsertFastMany
  • JobInsertFastManyNoReturning
  • JobInsertFullMany
  • MigrationInsertMany
  • MigrationInsertManyAssumingMain

Back when I was first implementing SQLite, the sqlc support was so
catastrophically bad that many, many basic utilities just could not be
used in queries. I tried like a half dozen different ways, but couldn't
find anything that was both supported by SQLite and sqlc, so I ended up
making a number of batch inserts into single inserts that get looped
over in the driver. This isn't as bad in SQLite because usually your
inserting to a local database (i.e. no network connections), but it's
still not ideal in that it's probably somewhat slower, and deviates from
the Postgres implementation.

Sqlc's made some SQLite improvements since then, and I found that I'm
now able to get a batch insert working `json_each` and `json_extract`.

Here, go through the existing SQLite driver function and update things
to use batch inserts wherever possible. Fixed operations:

* `JobInsertFastMany`
* `JobInsertFastManyNoReturning`
* `JobInsertFullMany`
* `MigrationInsertMany`
* `MigrationInsertManyAssumingMain`
@brandur brandur force-pushed the brandur-sqlite-batch-inserts branch from f267258 to 9937980 Compare June 5, 2026 16:29
@brandur brandur requested a review from bgentry June 5, 2026 17:17
@brandur
Copy link
Copy Markdown
Contributor Author

brandur commented Jun 5, 2026

@bgentry This should be strictly an improvement. The driver tests are pretty good so we should have pretty confidence that it all works as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant