hypequery insertion support

Some thoughts on implementation and rollout:

  • Phase 1: Define Scope

- Support typed row inserts into existing ClickHouse tables.

- Start with JSONEachRow only.

- Exclude schema migrations, table creation, INSERT ... SELECT, and bulk file ingestion from v1.

  • Phase 2: Add Adapter Support

- Extend the internal database adapter with an insert capability.

- Implement it in the ClickHouse adapter using the native ClickHouse client’s insert API.

- Preserve existing query, stream, raw SQL, and cache behavior unchanged.

  • Phase 3: Add Public API

- Add a direct db.insert(table, rows, options) API.

- Accept either a single row or an array of rows.

- Support per-insert options like queryId and ClickHouse settings.

- Keep insertion separate from the existing select query builder.

  • Phase 4: Add Type Safety

- Reuse existing schema inference so insert rows are typed from table columns.

- Validate table names at compile time.

- Validate row field names and value types at compile time.

- Decide whether v1 rows are strict or partially optional.

  • Phase 5: Improve Generated Types

- Enrich schema generation with ClickHouse column metadata.

- Detect nullable columns, defaults, materialized columns, alias columns, and generated expressions.

- Use that metadata to make insert row types more accurate.

- Mark server-generated columns as omitted or readonly for inserts.

  • Phase 6: Runtime Guards

- Throw a clear error if an adapter does not support inserts.

- Normalize single-row inserts into arrays.

- Reject empty row arrays unless there is a deliberate no-op policy.

- Add useful logging around insert execution and failures.

  • Phase 7: Tests

- Unit test adapter insert forwarding.

- Unit test public db.insert() behavior.

- Type test valid and invalid insert rows.

- Integration test insert rows into a temporary ClickHouse table and read them back with the existing query builder.

  • Phase 8: Documentation

- Document simple row insertion.

- Document batching guidance.

- Document supported formats and current limitations.

- Add examples for generated schemas and hand-written schemas.

  • Phase 9: Advanced Insert Support

- Add optional support for custom insert formats.

- Consider insertFromSelect or raw insert helpers.

- Consider batch helpers for high-volume ingestion.

- Consider dataset-level write APIs only after core insert support is stable.

Share update with 0 linked conversations as well

Upvoters
Status

Completed

Board
💡

Feature Request

Date

About 1 month ago

Author

Luke

Subscribe to post

Get notified by email when there are changes.