HTTP

omniload reads a file addressed by an HTTP or HTTPS URL, using the same reader stack as every other filesystem source. See that page for the supported formats, format and reader hints, and type normalization.

URI format

The URI is the URL of the file, exactly as you would open it.

https://example.org/path/to/data.csv
http://example.org/path/to/data.parquet

The query string is part of the address

Unlike the other schemes in this family, an HTTP URL’s query string is not connection configuration: it is sent to the server. That is what makes a presigned URL work, since the signature lives in the query and is computed over its encoded form, and an escape that carries meaning (%2F) is preserved rather than decoded. An escape of an unreserved character is normalized (%7E becomes ~), which every canonical signing scheme treats as the same value:

omniload ingest \
    --source-uri   'https://bucket.s3.amazonaws.com/exports/orders.csv?X-Amz-Signature=...&X-Amz-Expires=900' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.orders'

Quote the URI in your shell: & would otherwise background the command.

Note

Because the query is addressing information, connection options cannot ride in it, the way they do for s3:// and the other schemes. They are available when you build the source yourself, where a keyword argument reaches the underlying fsspec HTTP filesystem:

from dlt_filesystem.source.fsspec.http import HttpFilesystemSource

source = HttpFilesystemSource().dlt_source(
    "https://example.org/data.jsonl", "", block_size=65536
)

omniload ingest and run_ingest take no option of their own for these, but fsspec’s own environment configuration reaches the same constructor from either: FSSPEC_HTTP_BLOCK_SIZE=65536 for a scalar, or FSSPEC_HTTP holding a JSON object for anything structured. client_kwargs is the one exception, because this source always passes its own.

A block size of 0 is answered with the whole body in one request rather than with fsspec’s streaming file, whichever route sets it. That file cannot seek and reports seekable() as True anyway, so the readers that seek (Parquet, ORC, Feather, JSONL and a headerless CSV) failed on it rather than on the open. The cost is memory, and only for BSON and MessagePack: they were the two readers that worked on the streaming handle by reading it forward in pieces, and they now hold the whole body instead. Every other reader either seeks or reads the body in one call already.

Authentication

Credentials in the URL’s userinfo become an HTTP basic-authentication header. Percent-encode any character that a URI cannot carry literally, @ and / included:

https://<USERNAME>:<PASSWORD>@example.org/private/data.csv

Environment proxy settings and .netrc are honoured.

Extended type support

Type	Support	Remarks
Formats	✅	See filesystem.
Ranges	✅	Used when the server supports them.
Globs	✅	Requires an HTML directory index.
Last modified	✅	Requires valid Last-Modified on each file.

Examples

Load a public CSV file into DuckDB

omniload ingest \
    --source-uri   'https://example.org/path/to/data.csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'

Name the format for a URL that has no useful extension

An API endpoint rarely ends in .csv. Append a #format fragment:

omniload ingest \
    --source-uri   'https://api.example.org/exports/latest#csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'

Directory indexes and wildcards

A server that exposes an HTML directory index can be read with a glob. * selects links in one directory, while ** follows linked subdirectories recursively:

omniload ingest \
    --source-uri   'https://example.org/exports/**/*.csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.exports'

The server’s index defines what can be discovered. A wildcard cannot find files that the index does not link.

Incremental file selection

Add --filesystem-incremental to append rows only from new or modified files, as described on the filesystem page. Each selected file must return a valid Last-Modified header. A directory index does not carry that metadata, so an incremental glob makes one metadata request per matched file. A missing or malformed header stops extraction and names the query-free file URL. A plain, non-incremental load still works when the header is absent.

How much is transferred

Where the server honours byte ranges, the file is read in ranges rather than downloaded whole, so a line-delimited document starts producing rows before all of it has arrived. Two cases are read whole instead, because they cannot be read any other way:

• a server that ignores Range and answers with the entire body;

• a response with no Content-Length (ordinary Transfer-Encoding: chunked), which leaves the file with no known size and therefore no seekable form.

Parquet is read whole in every case: pyarrow asks for a file’s entire data section in one read, independently of the transport.

Limitations

• Read only. There is no portable way to write a file over plain HTTP.

See the WebDAV source for an HTTP-based transport with a defined listing protocol.