Reading Parquet from scratch in Swift

I work with Parquet files all the time — some on my disk, some in S3, some in R2, some sitting behind portal logins whose access keys I can never quite remember. After the Nth time juggling credentials to spot-check a Delta table, I gave up and started writing a viewer. The first thing it had to do was read the format from scratch in Swift. No Arrow, no parquet-cpp, no Python bridge — a single SwiftPM target that opens a .parquet file and hands back rows.

Here’s what that took.

Why not Arrow or parquet-cpp?

The obvious move is to wrap an existing C++ implementation. Apache Arrow’s parquet-cpp is battle-tested, fast, and supports every encoding the spec defines. It’s also a 200-megabyte build dependency, has its own memory model, requires a C++ bridge that fights ARC, and ships with a license file longer than most of my Swift code. For a macOS app distributed through a DMG, the wrong move.

The Parquet format has a reputation for being intimidating, but most of that intimidation is documentation density rather than implementation difficulty. The actual on-disk layout is small enough to fit in your head once you’ve walked it end-to-end. What follows is that walk.

The file layout: read it end-first

Open any .parquet file in a hex viewer and look at the last four bytes:

50 41 52 31   →   "PAR1"

That’s the magic trailer. The four bytes immediately before it are a little-endian uint32 giving the byte-length of the footer — a Thrift-encoded FileMetaData struct that describes the entire file: schema, row groups, column chunks, statistics, the works. The file’s actual data lives in column chunks earlier in the file, but you can’t make sense of any of it without the footer first.

So the read order is inside-out:

┌─────────────────────────────────────────────────────────────┐
│ PAR1                                                        │  ← leading magic
├─────────────────────────────────────────────────────────────┤
│ Row group 0                                                 │
│   Column chunk: customer_id                                 │
│   Column chunk: order_total                                 │
│   …                                                         │
├─────────────────────────────────────────────────────────────┤
│ Row group 1                                                 │
│   …                                                         │
├─────────────────────────────────────────────────────────────┤
│ Thrift-encoded FileMetaData (variable length)               │  ← step 3
├─────────────────────────────────────────────────────────────┤
│ footer length (uint32 LE)         │  PAR1                   │  ← steps 1-2
└─────────────────────────────────────────────────────────────┘

In Quarry that’s three small reads:

let trailer = try await reader.readSuffix(8)
guard trailer.suffix(4) == Self.magic else {
    throw ParquetFormatError.badMagic
}
let footerLen = trailer.withUnsafeBytes { raw -> UInt32 in
    raw.loadUnaligned(as: UInt32.self).littleEndian
}
let fileSize = try await reader.contentLength()
let footerStart = fileSize - 8 - Int64(footerLen)
let footerBytes = try await reader.read(range: footerStart..<(fileSize - 8))

That’s the whole “open a Parquet file” dance. The remaining work is all inside footerBytes.

One nice property of this layout: it’s friendly to cloud reads. The RandomAccessByteReader abstraction over local files and HTTP range requests means Quarry opens an S3 object in exactly two requests — one 8-byte suffix grab, one footer range — and never downloads the full file unless you actually scroll into row data. A 12 GB file in R2 opens in under a second on a flaky hotel Wi-Fi.

Thrift compact protocol in one file

The footer is encoded using Apache Thrift’s CompactProtocol. This sounds scary if you’ve never touched Thrift before, but the format is genuinely small. Five primitive operations and you can decode the entire spec:

Varints — ULEB128 unsigned integers. Read bytes until you see one with the high bit clear; each byte contributes 7 bits to the result.
Zigzag signed integers — (u >> 1) ^ -(u & 1). Stores small negative numbers in few bytes.
Field headers — one byte: high nibble is a delta against the previous field ID in this struct (a delta of 0 means “long form follows as a varint”); low nibble is a wire type.
Stop sentinel — a zero byte ends a struct.
Lists / binary / strings — varint length prefix, then payload.

That’s it. The entire CompactProtocol reader in Quarry is one ~230-line file with no dependencies. The field-delta trick is the cute part: instead of writing the absolute field ID for every field, Thrift just records how far the ID moved from the previous one. Since Parquet’s structs declare fields with mostly-contiguous IDs (1, 2, 3, 4, …), most field headers fit in a single byte.

Here’s the field-header read in full:

public mutating func readFieldHeader() throws -> FieldHeader {
    let b = try readByte()
    if b == 0 { return FieldHeader(id: 0, type: .stop) }
    let typeNibble = b & 0x0F
    let type = WireType(rawValue: typeNibble)!
    let deltaNibble = (b >> 4) & 0x0F
    let id: Int16
    if deltaNibble == 0 {
        // Long form: explicit zigzag varint field id.
        let raw = try readVarintI32()
        id = Int16(truncatingIfNeeded: raw)
    } else {
        let prev = fieldIdStack[fieldIdStack.count - 1]
        id = prev &+ Int16(deltaNibble)
    }
    fieldIdStack[fieldIdStack.count - 1] = id
    return FieldHeader(id: id, type: type)
}

The fieldIdStack tracks “last field ID seen” per nesting depth, because delta IDs reset on entering a nested struct. That stack — one Swift array of Int16 — is the only state CompactProtocol needs.

The other thing Thrift gives you for free is forward compatibility: if the format adds a new field tomorrow, you skip past it by wire-type without caring what it means. skip(type:) handles every wire type, and Quarry’s struct decoders call it for any field ID they don’t recognise. That’s how the same code reads files written by pandas 1.x, DuckDB, Spark 3.5, and Delta Lake table-format writers without special-casing any of them.

From a flat list to a schema tree

Once you’ve decoded FileMetaData, the schema arrives as a flat array of SchemaElement in depth-first order, with each element saying how many children it has. That’s it — no parent pointers, no explicit nesting. Reconstructing the tree is a one-pass recursive descent:

private static func parse(
    elements: [SchemaElement],
    index: inout Int,
    path: [String],
    parentDefLevel: Int,
    parentRepLevel: Int,
    isRoot: Bool
) throws -> SchemaNode {
    let e = elements[index]
    index += 1
    let childCount = Int(e.numChildren ?? 0)
    var children: [SchemaNode] = []
    for _ in 0..<childCount {
        children.append(try parse(elements: elements,
                                  index: &index,
                                  path: path + [e.name],
                                  …))
    }
    return SchemaNode(name: e.name, children: children, …)
}

The interesting work isn’t the recursion — it’s computing each node’s definition level and repetition level as you descend. These two numbers are how Parquet encodes nulls and nested structure without per-row null bitmaps.

Definition level increments by 1 each time you cross an OPTIONAL or REPEATED boundary. A value’s runtime def-level tells you “how many of my optional/repeated ancestors were actually present” — so a leaf with maxDefinitionLevel = 3 and a row whose def-level is 2 means one of its ancestors was null.
Repetition level increments only on REPEATED boundaries. It’s used to reconstruct list boundaries: a value’s runtime rep-level says “at what level of the nested list structure did a new element start.”

Storing these as maxDefinitionLevel and maxRepetitionLevel on every SchemaNode at parse time means the column reader doesn’t have to walk the tree at decode time — it just reads node.maxDefinitionLevel to know how many bits each definition-level entry needs. (RLE encoding of levels uses ceil(log2(maxLevel + 1)) bits per value.)

Reading a column: pages, encodings, the dictionary trick

Now the schema is in memory. To read a column, you look up the ColumnChunkInfo in RowGroup.columns, seek to its fileOffset, and start reading pages. Each page is one PageHeader (Thrift) followed by compressedPageSize bytes of payload. Page types you care about:

DICTIONARY_PAGE — present if the column uses dictionary encoding. Contains a list of distinct values in PLAIN encoding.
DATA_PAGE / DATA_PAGE_V2 — actual row values, possibly preceded by definition levels (one per row, including for nulls) and repetition levels (one per row, for nested data).

Once you’ve got the page bytes, decompress them. Quarry uses Apple’s Compression framework for ZSTD, LZ4_RAW, and ZLIB, plus a ~30-line C wrapper around the vendored Snappy. UNCOMPRESSED is a passthrough.

What’s left is decoding. Parquet defines a handful of encodings:

Encoding	Used for
PLAIN	Raw values, little-endian. The simple default.
RLE	Definition/repetition levels; boolean values. Hybrid run-length + bit-packed.
PLAIN_DICTIONARY / RLE_DICTIONARY	Indices into a dictionary page.
DELTA_BINARY_PACKED	Integer deltas, bit-packed in blocks.
DELTA_BYTE_ARRAY / DELTA_LENGTH_BYTE_ARRAY	Byte arrays with shared prefixes / explicit lengths.
BYTE_STREAM_SPLIT	Floats/doubles split into byte planes for better compression.

Quarry implements all of these in Sources/ParquetCore/Encoding/. None takes more than ~140 lines. The PLAIN decoder is exactly what it sounds like — read N bytes, interpret as the physical type. The RLE decoder is the bit-packing one that needs the most care: values are packed LSB-first within each byte, and the format alternates between RLE runs (“the next K values are all X”) and bit-packed groups of 8.

But the surprisingly clever bit is dictionary encoding. Most real-world columns have low cardinality — think string columns of customer types, country codes, status flags. Storing each value once in a dictionary page and writing indices into the data page can compress a 10-million-row column with 50 distinct values from 500 MB to under 10 MB before a general-purpose compressor like ZSTD even gets involved. The data page becomes a stream of small integers (often fitting in 6–8 bits per value once you bit-pack the indices), which then compresses extraordinarily well because runs are common.

The decode path is the dual: read the dictionary page once, decode it with PLAIN, then iterate the data page as RLE-encoded indices and look each one up in the dictionary array. A few dozen lines of Swift:

case .rleDictionary, .plainDictionary:
    let indices = try RLEDecoder.decodeBitPackedHybrid(
        bytes: dataBytes,
        bitWidth: bitWidth,
        numValues: numValues
    )
    return indices.map { i in
        guard i < dict.count else { throw … }
        return dict[i]
    }

That’s it. That’s the whole “fast column scan” story for the common case.

What “no dependencies” actually buys you

Three things, in order of how often I notice them:

The binary stays small. Quarry’s universal .app is under 5 MB. A parquet-cpp build would have been 80+ before app signing.
The build is fast. swift build from a clean checkout is under 30 seconds on an M1. No CMake, no submodules, no version pins to chase.
The format is no longer mysterious. When a user files a bug (“this file from $tool opens crooked”), I can step into a ThriftReader in the debugger and walk to the byte that lied. With a wrapped C++ lib, that bug becomes a stack frame in a foreign library and a Stack Overflow search.

What you give up is the parts of the spec that I don’t need: bloom filters, the page index for predicate push-down, some of the more exotic encodings that nobody seems to actually write. If a file in the wild uses them, Quarry surfaces an “unsupported encoding” message in the cell rather than crashing — and so far the only encoding I’ve actually seen in user files and not implemented is DELTA_BINARY_PACKED for byte arrays, which is rare enough that I haven’t been pushed to add it.

The rest of the format — the actual on-disk layout, the Thrift framing, the schema reconstruction, the seven encodings — is maybe 1,500 lines of Swift. If you can read footer-first, varints, and a recursive descent, you can read Parquet from scratch in any language. The “library” version is a convenience, not a requirement.