Package dev.vortex.api

Class DataSource

java.lang.Object

dev.vortex.api.DataSource

All Implemented Interfaces:

AutoCloseable

public final class DataSource
extends Object
implements AutoCloseable

A set of Vortex files opened through a Session. Data sources are cheap to open (only the first file is read eagerly, to determine the schema) and can be scanned multiple times.

Call close() to release this handle and its cached file resources promptly. Scans and partitions already created from it remain valid and retain the resources they still need. VortexCleaner releases an unclosed handle when it becomes unreachable.

Native calls are synchronized with close() to prevent another thread from freeing the handle during an operation.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	DataSource.ByteSize	Precision-aware byte size.
static interface	DataSource.RowCount	Precision-aware row count.

Method Summary

Modifier and Type	Method	Description
org.apache.arrow.vector.types.pojo.Schema	arrowSchema(org.apache.arrow.memory.BufferAllocator allocator)	Arrow schema of the data source (and of scans produced from it).
DataSource.ByteSize	byteSize()	Sum of the on-storage byte sizes of all files included in this data source along with the precision of that estimate.
void	close()	Releases this handle once; existing scans, partitions, and batches remain valid.
static DataSource	open(Session session, NativeReadable readable)	Open a single file through a caller-provided byte source.
static DataSource	open(Session session, String uri)	Open a single URI.
static DataSource	open(Session session, String uri, Map<String,String> properties)	Open one or more URIs or globs.
static DataSource	open(Session session, List<NativeReadable> readables)	Open files through caller-provided byte sources with the default read concurrency.
static DataSource	open(Session session, List<NativeReadable> readables, int readConcurrency)	Open one or more files through caller-provided byte sources instead of native storage clients.
static DataSource	open(Session session, List<String> uris, Map<String,String> properties)	Open one or more URIs or globs.
DataSource.RowCount	rowCount()	Row count along with the precision of that estimate.
Scan	scan(ScanOptions options)	Submit a scan.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

open

public static DataSource open(Session session,
 String uri)

Open a single URI.

open

public static DataSource open(Session session,
 String uri,
 Map<String,String> properties)

Open one or more URIs or globs. When a glob is used, the first match is opened eagerly; subsequent matches are opened lazily on scan.

Parameters:

session - open session

uri - single URI or glob

properties - object-store credentials / options

open

public static DataSource open(Session session,
 List<String> uris,
 Map<String,String> properties)

Open one or more URIs or globs. When a glob is used, the first match is opened eagerly; subsequent matches are opened lazily on scan.

Parameters:

session - open session

uris - URIs or globs to scan

properties - object-store credentials / options

open

public static DataSource open(Session session,
 NativeReadable readable)

Open a single file through a caller-provided byte source. See open(Session, List, int).

open

public static DataSource open(Session session,
 List<NativeReadable> readables)

Open files through caller-provided byte sources with the default read concurrency.

open

public static DataSource open(Session session,
 List<NativeReadable> readables,
 int readConcurrency)

Open one or more files through caller-provided byte sources instead of native storage clients. Every read the scan performs becomes an upcall into the corresponding NativeReadable, so this is the integration point for external I/O abstractions (for example Iceberg's FileIO). Each file is identified by NativeReadable.name(), which must be unique within readables.

The readables must remain open until this data source is closed and every derived scan, partition, and Arrow reader has finished reading. Native code never closes them.

Parameters:

session - open session

readables - byte sources

readConcurrency - maximum in-flight readFully calls across all files of this data source; 0 selects the default. Each in-flight read occupies one native thread and typically one stream on its readable.

arrowSchema

public org.apache.arrow.vector.types.pojo.Schema arrowSchema(org.apache.arrow.memory.BufferAllocator allocator)

Arrow schema of the data source (and of scans produced from it).

rowCount

public DataSource.RowCount rowCount()

Row count along with the precision of that estimate. Mirrors the Rust Precision<u64> returned by DataSource::row_count: DataSource.RowCount.Unknown when no estimate is available, DataSource.RowCount.Estimate for an inexact hint, DataSource.RowCount.Exact when the count is authoritative.

byteSize

public DataSource.ByteSize byteSize()

Sum of the on-storage byte sizes of all files included in this data source along with the precision of that estimate. Mirrors the Rust Option<Precision<u64>> returned by DataSource::byte_size: DataSource.ByteSize.Unknown when no estimate is available (for example when the filesystem listing did not return sizes), DataSource.ByteSize.Estimate for an inexact hint (some files contribute extrapolated sizes), and DataSource.ByteSize.Exact when every file has a known size.

scan

public Scan scan(ScanOptions options)

Submit a scan.

close

public void close()

Releases this handle once; existing scans, partitions, and batches remain valid.

Specified by:

close in interface AutoCloseable