Skip to content

N5

icechunk-n5 lets N5 read and write an icechunk repository. Its IcechunkKeyValueAccess is an N5 KeyValueAccess over an icechunk session, so n5-zarr's Zarr v3 reader and writer work on a repository as they do on a directory or a bucket.

  • From code, build a reader or writer on a session, write, and commit (Write and commit).
  • From a URL, icechunk-n5-universe lets N5Factory, and so Fiji and BigDataViewer, open URLs such as s3://bucket/repo|icechunk://branch.main/em/raw (Open from a URL).
  • icechunk-n5-codecs adds the pcodec and zlib codecs, which n5-zarr lacks (numcodecs codecs).
  • A few KeyValueAccess operations behave differently than on a file system (How icechunk differs from a file system).

For Fiji setup, see Use in Fiji.

Experimental

The API will change, and opening a repository from a URL needs an unreleased n5-universe.

icechunk in N5 terms

An icechunk repository holds a Zarr v3 hierarchy with version control on top. The icechunk docs cover these in depth.

  • A snapshot is one committed version of the whole hierarchy, with an ID such as 8AFVAQM5BQKKPY0GFGPG.
  • A branch names the latest snapshot of a line of work, as in git. A tag names a fixed snapshot.
  • A session reads one version, or collects writes on a branch. Its writes stay invisible to others until you commit, which makes a new snapshot atomically. A commit that loses a race with another writer fails with ConflictException and leaves the repository unchanged.
  • A session's store (session.store()) is its Zarr key/value view, which IcechunkKeyValueAccess reads and writes.

N5 sees one session as one container: every reader or writer reads one version, and a writer's chunks become a snapshot only when the session commits. The data is ordinary Zarr v3, which icechunk-python and zarr-python read, including gzip-compressed and sharded datasets.

Get the jars

Nothing is on Maven Central or the SciJava repository yet. Build from source with pixi and rustup, and install into your local Maven repository. In a clone of icechunk-java:

pixi run build-native
pixi run mvn -B install -DskipTests -pl icechunk-java,icechunk-n5,icechunk-n5-codecs -am

Then depend on icechunk-n5:

<dependency>
  <groupId>io.earthmover.icechunk</groupId>
  <artifactId>icechunk-n5</artifactId>
  <version>0.1.0-SNAPSHOT</version>
</dependency>
  • n5 is provided, and n5-zarr and n5-universe are not included, so add the versions you use from the SciJava Maven repository. icechunk-n5 builds against n5 4.0.2, n5-zarr 2.0.2 and n5-universe 3.1.0.
  • The jars target Java 8, like n5-universe and n5-ij.
  • A source build does not bundle the native library, so point the JVM at it: -Dicechunk.native.dir=/path/to/icechunk-java/native/target/debug.

The samples below are sections of examples/src/main/java/io/earthmover/icechunk/examples/N5Basics.java. Run it with pixi run example N5Basics.

Open a repository

A repository lives in a Storage: a local directory, memory, S3, Google Cloud Storage, Azure, or read-only HTTP.

try (Storage storage = Storage.localFilesystem(dir);
        Repository repo = Repository.create(storage)) {

Repository.open opens an existing one, and Repository.openOrCreate does either.

Write and commit

Open a writable session on a branch, build an n5-zarr writer on it, write, and commit:

try (Session session = repo.writableSession("main")) {
    N5Writer n5 =
            new ZarrV3KeyValueWriter(new IcechunkKeyValueAccess(session), "", new GsonBuilder(), false);
    n5.createGroup("em");
    n5.setAttribute("em", "units", "nm");
    DatasetAttributes attributes = n5.createDataset(
            "em/raw", new long[] {64, 64}, new int[] {32, 32}, DataType.UINT16, new GzipCompression());
    n5.writeChunk(
            "em/raw", attributes, new ShortArrayDataBlock(new int[] {32, 32}, new long[] {0, 0}, fill(1)));
    first = session.commit("Add em/raw");
}
  • The base path is "", so paths are relative to the repository root.
  • After commit the session is read-only. Open a new one for more changes, or keep a writer across commits.
  • Uncommitted writes never reach a snapshot. session.discardChanges() drops them.

Read a version

A read-only session opens the tip of a branch, a tag, a snapshot, or a branch as it was at a point in time. Two readers on two sessions see two versions of the same chunk:

try (Session latest = repo.readonlySession(Version.branch("main"));
        Session earlier = repo.readonlySession(Version.snapshot(first))) {
    N5Reader now =
            new ZarrV3KeyValueReader(new IcechunkKeyValueAccess(latest), "", new GsonBuilder(), false);
    N5Reader then =
            new ZarrV3KeyValueReader(new IcechunkKeyValueAccess(earlier), "", new GsonBuilder(), false);
    DatasetAttributes attributes = now.getDatasetAttributes("em/raw");
    short[] current =
            (short[]) now.readChunk("em/raw", attributes, 0, 0).getData();
    short[] original =
            (short[]) then.readChunk("em/raw", attributes, 0, 0).getData();
  • The other two kinds of version are Version.tag("v1") and Version.asOf("main", instant).
  • A session on Version.branch("main") stays on the tip it opened at. Open a new session to see later commits.

Open through n5-universe

N5Factory opens readers and writers over the same object. Pass StorageFormat.ZARR3, since icechunk holds Zarr v3 only.

try (Session session = repo.readonlySession(Version.branch("main"))) {
    N5Reader n5 = new N5Factory()
            .openReader(StorageFormat.ZARR3, new IcechunkKeyValueAccess(session), URI.create(""));
  • IcechunkKeyValueAccess ignores the URI's scheme and authority, so URI.create(""), / and icechunk://anything all open the repository root.
  • A caching factory such as N5FactoryWithCache, which Paintera's PainteraN5Factory extends, caches by URI. Give each repository its own URI, such as icechunk://<unique-id>.
  • The log line Compression 'org.janelia.saalfeldlab.n5.blosc.BloscCompression' could not be registered means n5-blosc's native library is missing. Other codecs are unaffected.

Open from a URL

icechunk-n5-universe lets N5Factory, and every tool that opens containers through it, open an icechunk URL.

N5Reader n5 = new N5Factory()
        .openReader("s3://icechunk-public-data/v1/era5_weatherbench2|icechunk://branch.main/1x721x1440");

Syntax

LOCATION|icechunk://VERSION/NODE

s3://bucket/repo|icechunk://branch.main/em/raw
gs://bucket/repo|icechunk://tag.v1
/data/repo|icechunk://GQQFH5G3AXKWZR5H33M0/labels
  • LOCATION: a local path, a file: URL, s3://bucket/prefix, gs://bucket/prefix, or an http(s):// URL.
  • VERSION: branch.NAME, tag.NAME or a snapshot ID. Without one, the URL opens the main branch.
  • NODE: an optional path inside the repository.

The syntax is the icechunk: scheme of the draft URL pipeline specification, which grew out of Zarr's ZEP 8 proposal. It is not yet an official Zarr standard and may change. Neuroglancer uses an earlier draft, icechunk:@branch.main, which icechunk-java rejects.

icechunk-java adds three things the specification does not have:

  • A location a RepositoryResolver claims, such as al:org/repo (see Resolvers).
  • A location ending in .icechunk needs no |icechunk: part, and opens the root of the main branch.
  • %7C is read as |, since URL fields percent-encode it.

URLs open read-only. openWriter on an icechunk URL throws, because N5 has no commit step and the writes would never become a snapshot. To write, build an IcechunkKeyValueAccess on a writable session, as in Write and commit.

What the provider does

  • Credentials. S3 and Google Cloud Storage open with the environment's credentials, such as the AWS default chain, and anonymously if that fails, so public repositories need none. An S3 bucket's region is looked up from the bucket.
  • Sessions. Each repository is opened once per location and kept open. Each URL opens a new session, so a branch is read at its tip when the URL is opened.
  • Virtual chunks. Every virtual chunk container the repository declares gets anonymous credentials, so no credentials go to a location the repository names. Local file system containers are never authorized. Set -Dicechunk.virtualChunks=none to authorize none.
  • Requests. A large read is split into parallel ranged requests of about 2 MiB, overriding the repository's own setting; -Dicechunk.requestSize sets another size in bytes. A failed request is tried up to 10 times, waiting from 100 ms up to 10 s between tries.

Resolvers

A RepositoryResolver opens a location that is not a storage URL, such as a repository name on a hosting service. It supplies the repository and its credentials, including those for virtual chunks. The provider finds resolvers through META-INF/services.

The separate icechunk-arraylake-java project provides one for Arraylake. It opens al:org/repo and https://app.earthmover.io/org/repo. Request sizes and retries apply as above.

Build it

The provider implements n5-universe's KeyValueAccessProvider interface, which no n5-universe release has yet. Install the n5-universe fork into your local Maven repository, as in Build the jars, then:

pixi run mvn -B install -DskipTests -Pn5-universe-provider -pl icechunk-n5-universe -am

N5Factory.openReader from code needs only that n5-universe change. Typing a URL into Fiji's dialog also needs an n5-ij change; Use in Fiji installs both, and Changes in other projects lists them.

numcodecs codecs

icechunk-n5-codecs lets n5-zarr read arrays that zarr-python wrote with numcodecs.pcodec or numcodecs.zlib, from icechunk or any other store. Put the jar on the classpath; n5 finds the codecs through the SciJava annotation index the jar carries.

  • numcodecs.pcodec replaces bytes as the array's serializer. It decodes uint16, int16, uint32, int32, uint64, int64, float32 and float64 through the icechunk native library. No float16, and no writing.
  • numcodecs.zlib is a compressor that stores each chunk as a zlib stream.

n5-zarr itself handles bytes, transpose, gzip, zstd, blosc and sharding_indexed. blosc needs n5-blosc's native library, which Fiji ships.

The module is experimental, until these codecs come from a codec package shared across N5 tools. Other numcodecs and imagecodecs codecs, such as LZW and JPEG 2000, are not available.

Keep a writer across commits

An application that commits often, such as an editor that commits each edit, would otherwise rebuild its readers and writers after every commit. IcechunkKeyValueAccess also takes a Supplier<Store>, which it asks for the store on each operation, so the application can swap in a new session after each commit:

AtomicReference<Session> current = new AtomicReference<>(repo.writableSession("main"));
N5Writer n5 = new ZarrV3KeyValueWriter(
        new IcechunkKeyValueAccess(() -> current.get().store()), "", new GsonBuilder(), false);
DatasetAttributes attributes = n5.getDatasetAttributes("em/raw");
for (int step = 0; step < 3; step++) {
    n5.writeChunk(
            "em/raw",
            attributes,
            new ShortArrayDataBlock(new int[] {32, 32}, new long[] {1, 1}, fill(10 + step)));
    current.get().commit("Edit " + step);
    current.getAndSet(repo.writableSession("main")).close();
}
current.get().close();
  • Close the old session only once no reads on it are in flight. A read that has started keeps its store, and fails if that session closes first.
  • Commit and swap from one thread. Between commit and the swap, the supplier still returns the committed, read-only session, so a write from another thread in that window throws.

How icechunk differs from a file system

icechunk stores groups and arrays, not files and directories, so a few KeyValueAccess operations behave differently:

  • Directories. A path is a directory if it is the root or holds a zarr.json. createDirectories does nothing: a group or array exists once its zarr.json is written.
  • Node types. A node's type is fixed: rewriting a group's zarr.json as an array's, or the reverse, fails.
  • Formats. The N5 and Zarr v2 formats cannot write to icechunk. A Zarr v2 key such as .zarray never exists, so n5-universe's format guessing finds Zarr v3.
  • Listings. On a read-only session, the first listing reads every node's zarr.json in one Store.listNodes call and keeps a copy. Later listings, isDirectory, existence checks for zarr.json, and zarr.json reads are answered from that copy. Opening one dataset by its path, with no listing, costs a few lookups. A writable session is asked on every call, since its writes would make a copy stale.
  • Errors. Errors from icechunk are thrown as N5IOException, with the IcechunkException as the cause, including a write through a read-only session. A path that is not a key icechunk can hold, such as a group's path, throws IllegalArgumentException.

What is missing

  • Opening from a URL needs forks of n5-universe and n5-ij, since no release has the changes yet (Changes in other projects).
  • n5 reads a shard's chunks one merged byte range at a time, each a separate call. icechunk can fetch many ranges in one call, but IcechunkKeyValueAccess does not use that yet.
  • Each write is copied twice on its way to icechunk, so peak heap is a few times the size of the largest shard.