# Changelog ## in progress - Connector: Verified support for DuckDB 2.x - Filesystem: An ORC export that PyArrow refuses, such as a time column under `--loader-file-format parquet`, no longer leaves a zero-byte file at the destination path or truncates a file already there. [dlt-filesystem] 0.22 is now the minimum version. - **Incremental strategies `upsert` and `cdc`.** `upsert` inserts and updates by primary key without deleting. `cdc` keeps the destination equal to the source, deleting the rows the source no longer holds; it is accepted for SQL database sources only, and like `scd2` it rejects options that read part of a table. Both need a primary key, declared by the table or named with `--primary-key`. dlt 1.31 is now the minimum version. `scd2` now also rejects `--interval-start` / `--interval-end` on a `query:` table that binds them, which windowed the read the same way. A plain `merge` or `delete+insert` now names the destination's default merge strategy, so a strategy an earlier load stored on the table, such as `cdc`, no longer carries over to it; a table loaded with `scd2` and then with a plain `merge` is merged with the destination's default and keeps no history for the keys that load reads. `upsert` and `cdc` are rejected up front on a destination that has no such strategy for the table. See [Incremental loading]. ## 2026/10/21 v0.20.0 - Keep datetime normalization compatible with dlt 1.31 while retaining support for older dlt releases. - Extract fragments of code into dedicated package [dlt-filesystem]. - **Notifications: a job report through Apprise when a run ends.** `--notify ` and `--notify-config `, both repeatable, send one short report per run, for its final outcome, to any target Apprise supports. Inline URLs receive every report, config entries route by their tags (`success`, `failure`, `always`), and `--notify-on failure` keeps a job quiet while it works. Every target is checked before the load starts, and a report that cannot be sent afterwards only logs a warning. The report names schemes, stages and exception types, never URIs, query text or error messages. Install `omniload[notify]`; `run_ingest` takes the same options as `notify`, `notify_config` and `notify_on`. See [Notifications]. - **Turso, read and written.** `turso:///path/to/app.db` is an in-process file and `turso://host?authToken=...` a remote database, Turso Cloud or a self-hosted `tursodb --sync-server` (`tls=false` for plain HTTP), each a source and a destination through pyturso's own SQLAlchemy dialects (`pip install 'omniload[turso]'`, not available on Windows). A remote database is read and written through a replica bootstrapped for the run and discarded after it; a destination run pushes the replica once the load has committed, and cannot be combined with `--pipelines-dir`. The destination writes to the `main` schema with `append` and `replace` only: a merge load into Turso deletes rows it should keep (tursodatabase/turso#6279), so `merge`, `delete+insert`, `scd2` and a source's own merge resources are refused before anything is written, as is `--full-refresh`. ## 2026/09/27 v0.19.0 - **Filesystem: `dlt-filesystem` declares what it imports.** A standalone `pip install dlt-filesystem` lacked Polars, so reading a CSV failed with `No module named 'polars'`. Polars is now a base dependency. BSON, the DuckDB CSV reader and the spreadsheet formats (XLSX, ODS) get the `bson`, `duckdb` and `spreadsheet` extras: without one, its formats are left out of the supported-formats message and their reader names the extra to install. `omniload` installs are unchanged, since omniload already carries all of these. - **Filesystem: XLSX files, written.** The local `file://` destination writes `.xlsx` workbooks with one worksheet named after the destination table, so a later load finds the table by that name. A value an Excel cell would store as a different value is refused, naming the column: an integer beyond 2^53, NaN or infinity, a string past 32,767 characters, a date before March 1900. Excel's own precision is documented rather than refused: floats keep 16 significant digits and times read back to the millisecond. Nested values, binary and decimals are written as text, as CSV writes them. ## 2026/09/23 v0.18.0 - **The filesystem and blob stack builds as its own distribution, `dlt-filesystem`.** It moves to `packages/dlt-filesystem/` and declares its own dependencies, so a dlt user can install the readers, the strict lister and the Arrow `readinto` shim without omniload's dependency list: 136 distributions against 250. Nothing an existing install line resolves to changes. `omniload`, `omniload[filesystem]`, `omniload[iterable]` and `omniload[full]` each resolve to the same set as before, because the extras now forward to the new distribution. The install hints a missing long-tail decoder prints name `dlt-filesystem[iterable]`, which is the extra that supplies it. - **Tests: the memory-store fixtures keep to their own store.** Two test modules subclass `MemoryFileSystem` to hold their fixtures apart from fsspec's process-global store, but `store` is a class attribute the subclass inherits rather than replaces, so their `clear()` emptied the store the remote-filesystem tests write into at import time. Each subclass now carries its own. - **Tests: the Arrow `readinto` shim's write-handle test reads the instance.** fsspec 2026.9.0 mirrors `readinto` onto `ArrowFile` for every mode, so `hasattr` no longer separates what fsspec provides from what the shim injects. The shim itself is unchanged and stays, because the supported floor is `fsspec>=2024.6`. - **Filesystem: Vortex files, read and write.** `.vortex` (and `.vortex.gz`) files read on every filesystem source, and the local `file://` destination writes them. A remote or gzipped file is staged to a local temporary file first, because the Vortex library opens files by path only. Opt-in through the new `vortex` extra, which needs Python 3.11 or newer and is included in `full`. ## 2026/09/17 v0.17.0 - **SQL sources: an `adbcbridge` backend, ADBC over ODBC.** `--sql-backend adbcbridge` reads a SQL source through the database's ODBC driver via [adbcBridge](https://github.com/singhpratech/adbcbridge) and yields Arrow record batches straight from the driver, with no Python row in between. SQLAlchemy still reflects the table and compiles the query, so `--incremental-key`, `--sql-limit`, `--sql-exclude-columns` and the `append` and `merge` strategies work as with the other backends. The ODBC connection string is derived from the source URI for PostgreSQL, Microsoft SQL Server, CrateDB (over its PostgreSQL wire protocol) and MySQL; `--sql-odbc-uri` gives it explicitly for anything else, and `OMNILOAD_ODBC_DRIVER_` renames a driver. Column types arrive at the precision the column declares (a PostgreSQL `TIMESTAMP(0)` as seconds). Installed with the `adbcbridge` extra (adbcBridge 0.1.3 or later, which keeps `timestamptz` instants on a server whose session zone is not UTC), part of `full`. Thanks, @singhpratech. - **Filesystem: an HTTP block size of `0` reads the body whole instead of handing back a file that cannot seek.** fsspec's streaming file reports `seekable()` as `True` and then raises on the first seek, so `block_size=0` broke every reader that seeks (Parquet, ORC, Feather, JSONL, and a headerless CSV given no column names to use) while CSV, JSON, BSON, MessagePack, CBOR and Avro went on working, so the failure read as format-specific rather than as a broken handle. It applies to the effective block size, so `FSSPEC_HTTP_BLOCK_SIZE=0` in the environment is covered as well as the keyword argument, which is what makes this reachable from `omniload ingest` rather than only from a source built in Python. - **Filesystem: the Parquet writer no longer emits an integer column its own reader cannot open.** Polars widens an integer past the signed 64-bit range to a 128-bit one, which Parquet has no type for: such a column was written as an untyped 16-byte `FIXED_LEN_BYTE_ARRAY`, which `pyarrow` refuses to read at all and DuckDB returns as a `BLOB`, so the export left carrying a value no consumer could spend. A column whose values all fit `0..2**64-1` is now narrowed to `uint64` and reads back unchanged, inside a struct or a list of structs as well as at the top level. The part of that band above `2**63` is where this writer is the alternative `write_orc` and `write_feather` are not, both of them refusing from there up. Past `2**64-1`, and on any negative in a column another row widened, all three refuse; this one names the offending value before the file is opened. - **Filesystem: a `dlt_source` implementation now declares the run options it consumes, instead of the run subtracting its own vocabulary from every call.** `filesystem_incremental` and `column_types` are named parameters on every reader in the family now, not names read out of `**kwargs`, so a caller passing something else in omniload's own run vocabulary (`page_size`, say) gets it forwarded straight to the fsspec/Arrow constructor rather than silently dropped. Most backends ignore an unknown keyword; at least one (WebDAV) raises. A row-level `--incremental-key` against one of these sources now raises for every scheme in the family (some silently accepted it before, since the guard was keyed on a value the caller had already cleared). ## 2026/09/10 v0.16.0 - **Filesystem: `filesystem()` and `readers()` accept every keyword dlt's own accept.** A call written against `dlt.sources.filesystem` that passes `kwargs`, `client_kwargs` or `incremental` by keyword now works against `dlt_filesystem.source.adapter` too, where it used to raise `TypeError`. `kwargs` and `client_kwargs` reach the fsspec constructor and its native client; an `incremental` cursor carrying `row_order` orders the listing by its cursor field, matching dlt for both `last_value_func` directions. The three are appended keyword-only rather than placed where dlt has them, so no existing positional call changes meaning; the flip side is that compatibility is by keyword, since in `filesystem()` dlt's positional slots for `kwargs` and `client_kwargs` are where `require_file_match` and `filesystem_incremental` sit. `extract_content`'s declared default is now `False`, which is what it has always resolved to and what dlt declares; a caller who wants file content in the listing passes `extract_content=True`, as before. Declaring `incremental` also makes dlt attach its `IncrementalResourceWrapper`, as it does to its own filesystem resource, so a caller reading `resource.incremental` gets the wrapper: the cursor is `resource.incremental.incremental`, and a resource with no cursor applied is that being `None` rather than `resource.incremental` itself. Bound clones of one resource now share a cursor, which is what dlt's do today rather than a guarantee; build separate resources where the cursors have to be independent. - **Filesystem: Apache Avro reads on every filesystem source.** `file://events/day.avro` loads, as does `s3://bucket/events/day.data#avro` and any other source that goes through the shared file readers. Avro is decoded with `polars`, which is already a core dependency, so nothing new is installed. Logical types come back as themselves: `date` as a date, `timestamp-micros` and `timestamp-millis` as UTC-aware datetimes, `time-micros` as a time, `decimal` as a decimal. `#columns=` and `#chunksize=` work as they do on the other columnar readers. Avro is read-only: the Avro writer available here corrupts a list column that holds an empty list before a non-empty one, so registering it would offer an export that fails silently on ordinary data. Three Avro schemas have no mapping and are rejected by name rather than loaded in part: a `map` field, a field typed `null` outright, and a union of more than two branches. - **Filesystem: `file://` writes CSV and Parquet through Polars.** Both writers built their column union by hand, because dlt omits a null key rather than writing it, and Polars builds that union itself from every row of the load. What the move changes is how a value is spelled, not which rows are written. A boolean writes as `true` / `false` in CSV instead of `True` / `False`, which is what the JSON and YAML writers already emit for the same value. A nested document writes as JSON (`{"a": 1}`) instead of as a Python repr (`{'a': 1}`), which no reader could parse, and binary as the base64 string the JSON writers use instead of as `b'hi'`. Under `--loader-file-format parquet`, the one load path that hands a writer native values, a time column carries nanosecond rather than microsecond precision, a datetime writes ISO 8601 with a `T` separator, and a Parquet decimal column is declared with the format's maximum precision rather than the source's, which leaves the values and their scale as they were. Some floats are spelled differently in CSV without changing value: `1e-05` writes as `0.00001`, `1e-07` as `1e-7`, and a NaN as `NaN`. CRLF line endings and Snappy Parquet compression are unchanged, and so are the Parquet column types apart from the time and decimal changes named above: a reader that reports Arrow types shows `large_string` where it showed `string`, which is the Arrow schema travelling alongside rather than the file's own, while the decimal annotation itself moves to the format's maximum precision and a wider fixed-length store. One case stops working: a Parquet export of a decimal column wider than 38 digits of precision, which PyArrow held in a 256-bit decimal and Polars cannot hold at all. The same column still exports to CSV, JSON, JSONL and YAML, digit for digit. - **Filesystem: Feather V2 reads on every filesystem source.** `file://events/day.feather` (or `.arrow`, or `.ipc`, or a `#feather` hint on any name) loads Apache Arrow IPC data through PyArrow, one record batch at a time, so a batched file is never materialized whole. `#columns=` and `#chunksize=` work as they do for ORC. Feather V1 is a different container and is refused by name rather than reported as a damaged file. - **Filesystem: `file://` writes Feather V2.** `file://export/users.feather` (or `.arrow`, or `.ipc`, or a `#feather` hint) writes one Arrow IPC file. Timezone-aware timestamps, times, decimals, nested lists and structs and all-null columns all reach the file as themselves when the writer is handed them. What a *load* delivers depends on dlt's staging rather than on the writer, and the filesystem page carries that table: the default staging writes temporal, decimal and binary columns as text, Parquet staging writes nested values as JSON text, a wholly null column is dropped either way, and a duration column fails at extract. That is `file://`-wide, not Feather's. - **Filesystem: the supported-format message names formats, not extensions.** A reader registered under several extensions listed all of them, so `.arrow` and `.ipc` would have read as separate formats alongside Feather. Only the canonical name is advertised now, matching what the write side already did for `.yml`. ## 2026/09/07 v0.15.0 - **Filesystem: `file://` reads back every intermediate format dlt stages.** The staged files were typed by running `file(1)` and matching its English output, which is absent on Windows, varies by version, and cannot tell gzipped CSV from gzipped JSONL because dlt compresses both. So `--loader-file-format csv` failed with a JSON decode error, and an uncompressed JSONL intermediate loaded or was refused as an unknown format depending on how many rows it held, since `file` names a one-line file differently from a many-line one. Both work now: the format comes from the name dlt gave the file, CSV is read with the delimiter, encoding and line terminator dlt wrote it with, and the staging directory names its own files rather than inheriting a `layout` or `extra_placeholders` meant for a user's real filesystem destinations. A format with no row-shaped reader, such as `insert_values`, is named in the error instead of being misread as JSON, and a destination table in dlt's reserved `_dlt_` namespace is refused rather than exported with dlt's own bookkeeping mixed into the rows. - **Filesystem: `file://` writes YAML.** `file://export/users.yaml` (or `.yml`, or a `#yaml` hint) writes one document holding a sequence, one mapping per row, which is the shape the `.yaml` reader expands into a table. Keys keep the column order the load produced, non-ASCII text is written as itself rather than escaped, and a load with no rows writes an empty sequence. Writing YAML needs PyYAML, the same `omniload[iterable]` extra that reading it needs. - **Filesystem: `file://` writes JSON, and the writers are registered rather than hand-wired.** `file://export/users.json` (or a `#json` hint) writes one JSON array document, which is the shape the `.json` reader expects. Registering a writer is now a single record in `dlt_filesystem.target.registry`, the way readers already work: the supported-format tuple, the error message and the end-to-end test matrix are all derived from it, and a test fails when the documentation names a different set. Two writers claiming one format is now an error instead of last-one-wins. - **Filesystem: written files are UTF-8 whatever the process locale is.** The readers decode as UTF-8 unconditionally, so a CSV or JSONL export containing non-ASCII text previously failed to write, or wrote bytes it could not read back, on a machine whose locale was not UTF-8. - Verified compatibility with Polars 2 - **CSV: `csv://` now shares the `file://` implementation.** It stays a CSV-only scheme, and every existing command still resolves, but reading and writing are the local filesystem connector's, so the behaviour changes. On read: values are typed rather than all-string (numbers and booleans are inferred; ISO date strings stay strings), a row whose fields are all empty is kept as a row of nulls instead of being dropped, an empty field loads as null when unquoted and as an empty string when quoted (the old reader dropped every empty field, so both produced null), and the path grammar gains globs, gzip, split form, Windows drive and UNC paths, and `#csv_headless` / `#csv_duckdb` hints. A rerun with no `--incremental-strategy` now **appends** rather than replacing; pass `--incremental-strategy replace` for the previous behaviour, which a destination that implements no append disposition now requires rather than merely recommends (MongoDB is one, and rejects the load without it). `merge`, `delete+insert` and `scd2` are rejected, because the shared reader exposes no incremental or merge key, and `--incremental-key` is rejected for the same reason: the row cursor it drove compared raw strings against parsed datetimes and crashed when given an interval, so it is removed rather than ported. File-level selection by modification time is available through `--filesystem-incremental`. A non-CSV selection (`csv://data.jsonl`, `csv://feed.dat#parquet`, `csv://book.xlsx`) is now rejected before any file is read; use `file://` for those. On write, a load that dlt split across several files wrote only the first of them and still exited zero, silently dropping the rest; every row now reaches the output file, in a deterministic but not necessarily source order. A destination path naming a different known format (`csv://out.jsonl`, `#parquet`) is rejected, while an extensionless or unrecognized-extension path (`csv://report`, `csv://out.dat`) still writes CSV. - Filesystem: added a reader for Apache ORC files. - Filesystem: added local ORC output through `file://`, implemented with PyArrow and preserving columns first encountered in later rows. ## 2026/09/01 v0.14.0 - Filesystem: Give the Arrow-backed filesystems (`file://`, `s3://`, `r2://`, `az://`, `hdfs://`, `rsync://`) read handles that implement `readinto`. fsspec's `ArrowFile` mirrors a fixed method list from the pyarrow stream it wraps and leaves `readinto` off it, so any reader that fills a caller-supplied buffer fails. Gzip is where that surfaces: fsspec registers isal's `IGzipFile` as its `gzip` codec when `isal` imports and the stdlib `GzipFile` only when it does not, and isal decompresses through `readinto`. A `.gz` file read through any of those schemes therefore raised `AttributeError: 'ArrowFile' object has no attribute 'readinto'` in an environment that merely carried the package, and read fine everywhere else. omniload declares no dependency on isal, but anything pulling `xopen` into the same environment brings it on x86-64 and AArch64. - Core: Add `--reshape :`, restructuring each source document before it is loaded. It flattens nested objects into columns, coerces types, and leaves arrays as real lists rather than opaque JSON. Three engines: `python::` (per row, preserves `Decimal`), `jq:` (per row, through Tikray) and `polars:` (per Arrow batch, compiled through macropipe). The jq and Polars engines need the new `omniload[reshape]` extra, and the Polars one needs an Arrow-yielding source, which today means MongoDB only. While a reshape is active the MongoDB source no longer json-hints top-level arrays, so they land as child tables rather than JSON columns. That last part is MongoDB-only: filesystem readers register `max_table_nesting = 0`, so a list a reshape produces there stays a JSON column. - MongoDB: Build a `PyMongoArrowContext` per wire batch in the Arrow loaders rather than once per cursor. `finish()` leaves the builders it accumulated in place, so the shared context replayed each earlier batch as a row of nulls: a collection spanning two batches emitted phantom all-null rows, and did so on any primary key. Reachable since `data_item_format` became settable per run. - Connector: Added source and target adapters for the Delta Lake table format ## 2026/08/27 v0.13.0 - mq-bridge: Require mq-bridge-py 0.4.7 (was 0.3.2) and add three transports: Redis Streams (`redis-streams+mqb://`), Apache Pulsar (`pulsar+mqb://`) and WebSocket (`websocket+mqb://`). Pulsar is not built into mq-bridge but ships as a separate native plugin, installed via the new `omniload[pulsar]` extra and registered on demand; it stays out of the base install (and out of `omniload[full]`) because its wheels require glibc >= 2.34 on x86_64, glibc >= 2.38 on aarch64, and omit Windows arm64, all narrower than omniload's own platform floor. mq-bridge names its Redis endpoint `redis_streams`, but a URI scheme cannot contain an underscore, so omniload addresses it as `redis-streams+mqb://`. Note that a WebSocket *source* is a server: mq-bridge binds the authority and ingests the frames clients push to it, so `--source-table` names the HTTP path served and the run is bounded by `max_messages`/`idle_timeout_ms` rather than by end-of-stream. - mq-bridge: Nack the batches left outstanding at the end of a run instead of only closing the consumer, so the broker redelivers them immediately rather than after a visibility timeout. This covers both a failed load and a `--yield-limit` that truncated a batch mid-yield. On transports with no per-message nack -- Kafka -- this still only leaves the offset unadvanced, so redelivery waits for the next run or rebalance as before. Delivery is unchanged otherwise -- still at-least-once, deduplicated on `_mqb_id`. - Filesystem: build reader transformers from registration records instead of a second hand-written adapter list, and expose the DuckDB CSV reader through `#csv_duckdb` and `.csv_duckdb`. - Connectors: Added support for Apache Pulsar, WebSockets, Redis streams. Thanks, @marcomq. - Connectors: Removed gRPC due to found issues. Thanks, @marcomq. ## 2026/08/24 v0.12.0 - HTTP: Support `--filesystem-incremental` when every selected file returns a valid `Last-Modified` header. A glob over an HTML directory index obtains the timestamp with one metadata request per matched file. A concrete file reuses the header already returned during discovery. A missing or malformed header stops extraction and names the query-free file URL instead of receiving dlt's synthesized current time and reloading on every run. - HTTP: Resolve `*`, `?` and recursive `**` wildcards against links exposed by a HTML directory index. The server's index remains the boundary of what the source can discover. - Table names: One quote-aware parser reads `catalog.schema.table` for every source and destination, replacing six ad-hoc conventions that each did something different with the same spelling. Names are right-aligned, so a shorter name keeps its shape: `table` and `schema.table` are still spelled the way they were, with the MySQL and SQLite routing changes below the exception to what they mean. Components may be quoted individually with `"`, `` ` `` or `[]`, which is how a name containing a dot is written; a whole path wrapped in one pair of backticks is refused by name, because each component quotes separately. Every platform declares the shape it accepts, so a name it cannot represent is rejected with a message that says what the platform takes and where to put the catalog instead, rather than being silently truncated or flattened. Thanks, @hampsterx. - Table names: A three-component name now reaches the destination's catalog. It sets `project_id` on BigQuery, `catalog` on Databricks, the database on MotherDuck, and `aws_data_catalog` on Athena, which had no way to reach a non-default catalog at all. For Postgres, Snowflake, Redshift, MSSQL, Synapse and Trino the catalog is the database the connection opens, so the first component selects it: `--dest-table reporting.public.orders` connects to `reporting` whatever the URI path said. That is a deliberate call, and the URI stays the way to pin one database for a run. DuckDB and CrateDB keep their two-level names, since neither can address a second database on one connection. Thanks, @hampsterx. - Table names: Three behaviour changes come with that. A dotted MySQL or SQLite destination name is now routed rather than flattened: `--dest-table foo.bar` on MySQL writes table `bar` in database `foo`, where it previously created one table whose name normalized from the literal string `foo.bar`, and SQLite's `main.widgets` no longer becomes a table called `main.widgets` in `main`. Both wrote to the wrong place before, so a pipeline relying on the old spelling loads somewhere new. And a three-part SQL *source* name is now refused rather than misparsed. It used to be cut at the first dot, so `public.order.items` silently became schema `public` and table `order.items`; SQL sources are two-level, so it now gets an error naming the accepted format and pointing at `--source-uri` for the catalog. A table whose own name contains a dot is spelled `public."order.items"`. Thanks, @hampsterx. ## 2026/08/21 v0.11.0 - Azure: Read `az://`, `adls://` and `abfss://` through `pyarrow.fs` instead of adlfs, for the per-file open cost every glob load pays. Measured against Azurite: opening and reading 300 small blobs costs 2.4 ms/blob through Arrow against 6.7 ms/blob through adlfs, and a 64 MB read runs at 456 MB/s against 193 MB/s. Listing a glob is one recursive request, recursive only when the pattern crosses a level. One client serves flat-namespace and Gen2 accounts, which it detects itself, and a custom `account_host` now yields both the Blob and the Data Lake endpoint so a Gen2 account behind a sovereign-cloud or emulator endpoint stays reachable. Every credential mode the connector accepts is carried over: account key, SAS token, service principal, and connection string, and a connection string's `DfsEndpoint` is honoured as given rather than derived. File-level incremental selection and its cursor identity are unchanged, so no cursor resets. Writing, and the staging download that serves a remote database file, continue to use adlfs. Thanks, @hampsterx. - Azure: Two behaviour changes come with that move. `api_version` is no longer accepted on a source URI and is rejected with a named error, because Arrow pins the API version its bundled SDK speaks and offers no override; an emulator that enforces the version check needs to be started with `--skipApiVersionCheck`. And a source connection string now has to name its account: one that identifies the account by endpoint alone (a bare `SharedAccessSignature` with a custom `BlobEndpoint` and no `AccountName`) is rejected, because the reader takes the storage account as the root of the filesystem. Thanks, @hampsterx. ## 2026/08/18 v0.10.1 - Azure: Scope file-level incremental cursors from connection-string sources to the parsed storage account and Blob endpoint. Different accounts using the same container and glob no longer share a cursor and silently skip files. Existing connection-string cursors reset once on upgrade, so the first run reads the selection again and `append` loads may duplicate rows. Use `--full-refresh` for that first upgraded load when duplicates are unacceptable. Connection strings continue to reach adlfs unchanged. ## 2026/08/15 v0.10.0 - HTTP: Serve `http://` and `https://` through the filesystem family, so an HTTP URL reads with the same formats, format hints and reader hints as every other transport. New: gzipped documents, whole-JSON documents, XML, YAML, msgpack, CBOR, BSON and workbooks; a `#format` or `#key=value` fragment on the URL; and, where the server serves byte ranges, reading in ranges instead of downloading the whole body first. A signed URL keeps working: the query string is treated as part of the address, so an escape that carries meaning (`%2F`) reaches the server preserved rather than decoded, and the query stays out of record identity, log lines, and the errors discovery and whole-body reads raise. A server that ignores `Range` and one that answers chunked with no `Content-Length` are read whole, as before. Thanks, @hampsterx. - HTTP: Four behaviour changes come with that move. A re-run with no `--incremental-strategy` now appends rather than replacing, matching the rest of the family, and an explicit `append` or `replace` is now honoured; `--incremental-key` is now rejected rather than ignored; `--filesystem-incremental` is refused, because an HTTP response need not carry a `Last-Modified` header and a missing one would read as "just now" and reload everything; and connection options can no longer ride the URI query string, since that is the address, so they are available only when building the source object directly, not through `omniload ingest` or `run_ingest`. Timeouts stay at 30 seconds and environment proxy and `.netrc` settings are still honoured. Thanks, @hampsterx. - Filesystem: Fixed `csv_headless`, which named columns with a Polars argument that *selects* columns instead. Reading a header-less CSV either failed outright or returned one shifted row, whether the names came from `--columns` or were generated. Thanks, @hampsterx. - Filesystem: A `#format` fragment on a source URI is now honoured, not only one on `--source-table`. Reader hints (`#key=value`) were already read from either. Thanks, @hampsterx. - Filesystem: Discovery no longer fails when a filesystem reports no size for a file, which any HTTP response without a `Content-Length` does. Thanks, @hampsterx. - Filesystem: Added a `json` format, so a `.json` file loads on every transport in the family. An object becomes one row and an array one row per element, whether or not the document is pretty-printed, and a `.json` file that holds line-delimited records still loads. `.jsonl` keeps its own strict one-record-per-line reader. A URI fragment of `#json` is now read as a format hint rather than as literal text. Thanks, @hampsterx. ## 2026/08/12 v0.9.2 - Filesystem: Started honouring `--filesystem-incremental` and `--columns` on every transport in the family. They were accepted and then ignored on the fourteen schemes that build their resource through the shared locator, among them `ftp://`, `smb://`, `oss://`, `oci://`, `webhdfs://`, `dropbox://`, `gdrive://`, `sharepoint://` and `dbfs://`. Thanks, @hampsterx. - Filesystem: Stopped passing omniload's own run parameters to fsspec filesystem constructors, which fragmented fsspec's instance cache and would be rejected by a backend that validates its keyword arguments. A source URI query parameter sharing a name with one of those run options is now ignored rather than forwarded; the run parameter already took precedence over it, so no connection setting changes meaning. Thanks, @hampsterx. ## 2026/08/05 v0.9.1 - Filesystem: List and read through every registered scheme, including `r2://`, `oss://`, `hdfs://`, `smb://`, `ftp://`, `dbfs://`, `oci://` and `webhdfs://`, by resolving a file's modification date from the listing the filesystem client returns. ## 2026/08/04 v0.9.0 - Database: Load SQLite and DuckDB databases that live on a remote filesystem, addressed by their plain object URI, for example `s3://analytics/snapshots/events.duckdb`. Thanks, @hampsterx. - Filesystem: Treat XLSX and ODS workbooks as multi-table sources by default, while preserving one-table loads through explicit worksheet selectors. Thanks, @hampsterx. - Filesystem: Refreshed S3 directory listings between repeated loads so file-level incremental runs detect newly uploaded objects. - Connectors: Added Azure Storage destination support for connection-string authentication and custom endpoints. - Connectors: Reject mixed Azure Storage credential modes instead of silently ignoring extra fields in source and destination URIs. - Filesystem: Fail ingestion when a concrete source path matches no file while keeping unmatched glob selections valid. Thanks, @hampsterx. - Filesystem: Added rsync source connector. Thanks, @oferchen. - Filesystem: Enabled loading multiple workbook worksheets as tables. Thanks, @hampsterx. - Filesystem: Enabled loading remote SQLite and DuckDB databases. Thanks, @hampsterx. ## 2026/07/27 v0.8.0 - Core: Added real [SCD2] loading when selecting the `scd2` strategy. Thanks, @hampsterx. - Filesystem: Added file-level incremental loading. Thanks, @hampsterx. - Filesystem: Migrated local filesystem access to Apache Arrow. - Filesystem: Added support for reading from Databricks, Dropbox, FTP, Google Drive, HDFS, OCI, OneDrive, OSS, R2, SharePoint, SMB, WebDAV, and WebHDFS. - Core: Applied a few bits of internal package refactoring. ## 2026/07/16 v0.7.0 - Connectors: Added Azure Blob Storage (`az://`) and Azure Data Lake Storage Gen2 (`adls://`, `abfss://`) source and destination adapters, backed by `adlfs`. Supports account-key, SAS-token, and service-principal auth. Thanks, @hampsterx. - Connectors: Fixed remote filesystem destinations (Azure Blob, GCS, S3) aborting at the end of a load because the required `post_load` hook was missing from the shared base class. Thanks, @hampsterx. - Filesystem: Added `#key=value` reader hint channel to source URIs to propagate options to source implementations. Thanks, @hampsterx. - Filesystem: Added readers for CBOR, MessagePack, XML and YAML formats. Thanks, @hampsterx. - Filesystem: Added readers for Excel workbook (XLSX) and OpenDocument spreadsheet (ODS) formats. - Filesystem: Fixed and documented semantics of append/replace strategy and append/full-refresh behaviour. Thanks, @hampsterx. - Dependencies: Updated to dlt 1.29 ## 2026/07/06 v0.6.0 - Connectors: Added `file://` source and destination adapters that read and write from/to local CSV / JSONL / Parquet files. Thanks, @hampsterx. - Connectors: Refactored filesystem internals - Connectors: Migrated filesystem CSV reader from pandas to Polars - Filesystem readers: Added support for BSON. Thanks, @hampsterx. ## 2026/07/02 v0.5.0 - Connectors: Updated to Asana client v5 - Core: Started using standard Python logger, removed `--quiet` option - API: Made option `--source-table` optional to prepare for streaming sources - Connectors: Added an mq-bridge source for streaming brokers (Kafka/NATS/AMQP/MQTT/ZeroMQ/IBM MQ/AWS SQS), via the `+mqb://` URI scheme, with dotted query keys (e.g. `tls.required=true`) for nested config such as TLS. Thanks, @marcomq. - Maintenance: Started using `orjson` across the board - Connectors: Centralize file-format to reader mapping across all sources. Thanks, @hampsterx. ## 2026/06/25 v0.4.0 - Maintenance: Refactored module namespace. If you are using omniload as a library, this introduces many breaking changes. However, the new layout is much more ergonomic. - Core: Added lazy-loading adapter module registry, to speed up startup times. ## 2026/06/24 v0.3.0 - Feature: Added embeddable `run_ingest()` Python API. Thanks, @hampsterx. - Dependencies: Updated to dlt v1.28 - Dependencies: Updated to clickhouse-connect v1 ## 2026/06/21 v0.2.0 - Kafka: Expanded `default_msg_processor` into a miniature decoding unit - dlt: Migrated from `ensure_pendulum_datetime` to `ensure_pendulum_datetime_utc` ## 2026/06/19 v0.1.0 - Dependencies: Permitted installation of SQLAlchemy 2.0 - Dependencies: Relaxed package requirements across the board - Documentation: Migrated from VitePress to Sphinx - Runtime: Used tqdm progress bar instead of spinner ## 2026/05/17 v0.0.0 - Imported code from [ingestr v0.14.155] - Project: Make it a library: Streamlined dependencies, now relaxed and inlined - Project: Migrated from Hatch to vanilla setuptools - Project: Migrated from Makefile to Poe the Poet - Project: Started using `versioningit` for versioning - Project: Standardized version handling - OCI: Modernized `Dockerfile` - Documentation: Trimmed README, copy editing - CI: Validated on Python 3.14 - OCI: Updated to Python 3.14 and Debian 13 "trixie" - dlt: Turned off telemetry due to excessive requests - Runtime: Removed interactive mode `--yes`, replaced with `--dry-run` - Packaging: Modernized PyPI and OCI publishing [dlt-filesystem]: https://github.com/dlt-contrib/dlt-filesystem [ingestr v0.14.155]: https://github.com/bruin-data/ingestr/tree/v0.14.155 [Incremental loading]: getting-started/incremental-loading.md [Notifications]: getting-started/notifications.md [SCD2]: https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row