Apache ORC: Optimized Row Columnar Format for Big Data
What Is Apache ORC?
Apache ORC (Optimized Row Columnar) is a self-describing, type-aware columnar file format for the Hadoop ecosystem. Originally developed by Hortonworks and Facebook in 2013 as a successor to the RCFile format, ORC was donated to the Apache Software Foundation and became a top-level project in 2015. It is the native format of Apache Hive and is also fully supported by Apache Spark, Presto, Trino, Flink, and most modern data lakehouse engines.
ORC competes directly with Apache Parquet in the columnar storage space. While Parquet is more commonly used in the Spark/Python/cloud ecosystem, ORC is often the better choice for Hive-heavy workloads and environments where ACID transactions on data lakes are required.
Files use the .orc extension and are fully self-describing: schema, statistics, compression, and data are all in the same file.
ORC vs. Parquet: Key Differences
| Feature | ORC | Parquet |
|---|---|---|
| Native format for | Apache Hive | Apache Spark / Impala |
| ACID transactions | Yes (Hive ACID) | No (requires Delta/Iceberg) |
| Compression default | ZLIB | Snappy |
| Bloom filters | Built-in | Plugin (requires explicit config) |
| Row-level statistics | Yes (per stripe) | Yes (per row group) |
| Predicate pushdown | Yes | Yes |
| Nested types | Yes (struct, list, map, union) | Yes (struct, list, map) |
| Union type | Yes | No |
| Schema evolution | Add columns | Add/rename columns |
| File format versions | ORC 1 (stable) | v1, v2 (Parquet2) |
| Typical size vs CSV | 70–85% smaller | 70–85% smaller |
| Cloud ecosystem | Good | Better (native S3 support) |
ORC File Structure
An ORC file is organized into stripes (analogous to Parquet's row groups):
ORC File:
├── Postscript ← file footer location, compression codec, ORC version
├── File Footer ← schema, stripe statistics, file-level stats
│ ├── Schema (TypeDescription) ← tree of types
│ ├── Stripe Information[] ← offset, length, row count for each stripe
│ └── File Statistics[] ← min/max/sum/count per column for entire file
├── Stripe 0 (default: 256 MB)
│ ├── Index Data ← row-level statistics per row group (10k rows)
│ │ └── Row Index[] ← min/max per column chunk of 10k rows
│ ├── Data ← encoded column data
│ │ ├── Column 0 (struct) ← presence stream (null bitmap)
│ │ ├── Column 1 (long) ← data stream + length stream
│ │ └── Column 2 (string) ← data stream + length stream + dictionary
│ └── Stripe Footer ← stream positions within stripe
└── Stripe 1 ...
Stripes
Stripes are the top-level unit of parallelism — each stripe is processed by one task in MapReduce/Spark. Default stripe size is 256 MB (vs Parquet's 128 MB row groups). Larger stripes mean better compression but less parallelism for small files.
Row Groups
Within each stripe, data is further subdivided into row index entries every 10,000 rows. These contain per-column min/max statistics used for skip row groups during predicate pushdown — ORC can skip individual 10k-row blocks within a stripe without reading their data.
Column Encodings
ORC chooses encodings per column based on data characteristics:
- DIRECT: raw values with an optional run-length encoding (RLE v2)
- DICTIONARY: dictionary-encoded strings (when cardinality is low)
- DIRECT_V2: RLE v2 with delta encoding for integers
- DICTIONARY_V2: dictionary with V2 encoding
RLE v2 (the ORC-specific encoding) supports:
- Short Repeat: repeated value sequences
- Direct: random values (bit-packed)
- Patched Base: values close to a base with outlier patches
- Delta: monotonically increasing/decreasing sequences (timestamps, sorted IDs)
Working with ORC in Python: pyorc
import pyorc
import io
# Define schema as a string
schema = "struct<id:int,name:string,age:int,salary:double,active:boolean>"
# Write ORC file
with open("employees.orc", "wb") as f:
with pyorc.Writer(f, schema, compression=pyorc.CompressionKind.ZSTD) as writer:
writer.write((1, "Alice", 30, 75000.0, True))
writer.write((2, "Bob", 25, 62000.0, True))
writer.write((3, "Carol", 35, 91000.0, False))
writer.write((4, "Dave", 28, 58000.0, True))
# Read ORC file
with open("employees.orc", "rb") as f:
reader = pyorc.Reader(f)
print(reader.schema) # struct<id:int,...>
print(reader.num_of_rows) # 4
for row in reader:
print(row) # (1, 'Alice', 30, 75000.0, True)
# Read as dict
reader2 = pyorc.Reader(open("employees.orc", "rb"),
struct_repr=pyorc.StructRepr.DICT)
for row in reader2:
print(row['name'], row['salary'])
With Apache Spark
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType, StringType, DoubleType
spark = SparkSession.builder.appName("ORC Demo").getOrCreate()
# Write DataFrame as ORC
df = spark.createDataFrame([
(1, "Alice", 30, 75000.0),
(2, "Bob", 25, 62000.0),
(3, "Carol", 35, 91000.0),
], ["id", "name", "age", "salary"])
df.write.mode("overwrite").orc("hdfs:///data/employees/")
# Read ORC
df2 = spark.read.orc("hdfs:///data/employees/")
# Predicate pushdown — ORC skips irrelevant stripes/row groups automatically
df2.filter("salary > 70000").select("name", "salary").show()
# Partitioned ORC write
df.write.mode("overwrite").partitionBy("age").orc("hdfs:///data/emp_partitioned/")
Hive ACID: ORC with Transactions
ORC is the only format that natively supports Hive ACID — full INSERT/UPDATE/DELETE on Hive tables stored in ORC:
-- Create transactional Hive table (requires ORC + ACID)
CREATE TABLE employees (
id INT,
name STRING,
salary DOUBLE,
dept STRING
)
CLUSTERED BY (id) INTO 4 BUCKETS
STORED AS ORC
TBLPROPERTIES ('transactional'='true');
-- Full DML support on ORC ACID tables
INSERT INTO employees VALUES (1, 'Alice', 75000, 'Engineering');
UPDATE employees SET salary = 80000 WHERE id = 1;
DELETE FROM employees WHERE dept = 'Legacy';
-- ACID ORC stores delete deltas as separate delta files
-- Compaction merges them back into base ORC files
ALTER TABLE employees COMPACT 'MAJOR';
Bloom Filters in ORC
ORC has built-in bloom filter support — a probabilistic data structure that tests whether a value is definitely absent or possibly present:
-- Enable bloom filters on frequently filtered columns
CREATE TABLE orders (
order_id BIGINT,
customer_id BIGINT,
product_sku STRING,
amount DOUBLE
)
STORED AS ORC
TBLPROPERTIES (
"orc.bloom.filter.columns" = "customer_id,product_sku",
"orc.bloom.filter.fpp" = "0.05" -- false positive probability: 5%
);
When querying WHERE customer_id = 12345, ORC checks each stripe's bloom filter. If the filter says the value is definitely absent, the entire stripe is skipped without reading data — even faster than min/max statistics alone (which only help for range queries on sorted data).
ORC Compression Options
| Codec | Speed | Ratio | Best for |
|---|---|---|---|
| NONE | Fastest write | No reduction | Hot data, debugging |
| SNAPPY | Fast R/W | ~40–50% smaller | Interactive queries |
| ZLIB | Medium | ~60–70% smaller | Cold storage (default) |
| ZSTD | Fast R + medium W | ~60–70% smaller | Recommended for new files |
| LZO | Fast R | ~45–55% smaller | Low-latency reads |
| LZ4 | Very fast R | ~45% smaller | Ultra-low latency |
| BROTLI | Slow | ~65–75% smaller | Maximum cold storage |
Converting Between Formats
# CSV → ORC via Hive
CREATE TABLE raw_csv (id INT, name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
LOAD DATA INPATH '/data/input.csv' INTO TABLE raw_csv;
CREATE TABLE orc_table STORED AS ORC AS SELECT * FROM raw_csv;
# Parquet → ORC via Spark
spark.read.parquet("input.parquet").write.orc("output.orc")
# ORC → Parquet
spark.read.orc("input.orc").write.parquet("output.parquet")
Practical Tips
- Use ORC when on Hive — it's the native format; all Hive optimizations (vectorization, ACID, bloom filters) work best with ORC
- Use Parquet when on Spark/Python — broader ecosystem support, better cloud-native tooling
- Enable bloom filters on high-cardinality columns used in equality filters (user_id, order_id, SKU) — they complement min/max stats which only help with range predicates
- ZSTD compression is the best modern choice: better ratio than ZLIB at comparable or better speed
- Stripe size 64–256 MB: smaller stripes for more parallelism on small clusters; larger stripes for better compression on large datasets
- ORC vectorization in Hive: set
hive.vectorized.execution.enabled=true— reads ORC in 1024-row batches using SIMD instructions, typically 3–5× faster than row-at-a-time processing - Check ORC file stats:
hive --orcfiledump -d employees.orcdumps structure, statistics, and bloom filter info
ORC's built-in ACID support, bloom filters, aggressive RLE v2 encoding, and deep Hive integration make it the best choice for Hadoop-native analytical workloads — particularly where transactional updates on data lake tables are required.
Related conversions
Frequent conversions across the catalogue: