Data Engineering
Standards in the catalog that belong to data engineering — 10 of them. A standard is listed here only when it is specific to this industry; the general-purpose ones it also relies on, like HTTP, OAuth and JSON Schema, deliberately carry no domain.
The same industry, as data models
5 data models define what information exists in data engineering, as opposed to the standards applied in API operations listed below. Same industry, two lenses.
See the data engineering data models →| Standard | What it is |
|---|---|
| Apache Arrow | Apache Arrow is a language-independent columnar memory format specification for flat and hierarchical data, optimized for analytics on modern hardw... |
| Apache Hudi | Apache Hudi is an open table format built around incremental processing — upserts, deletes and change streams over data lake tables. Where the othe... |
| Apache Iceberg | Apache Iceberg is an open table format for very large analytic datasets sitting on object storage. It adds a metadata layer above the data files — ... |
| Avro | Apache Avro is a data serialization system that provides compact binary encoding of structured data along with schema definitions, enabling efficie... |
| Data Modeling | Data modeling is the practice of defining the structure, relationships, and constraints of data so that information can be organized, stored, and a... |
| Data Warehouse Schemas | Organizational patterns for structuring data in warehouses, including star schema, snowflake schema, and galaxy schema designs that optimize query ... |
| Delta Lake | Delta Lake is an open table format and transaction protocol that brings ACID transactions, versioning and schema enforcement to data lakes. Origina... |
| Dimensional Modeling | A data modeling technique used in data warehousing that structures data into fact tables containing measurable metrics and dimension tables contain... |
| Lakehouse Architecture | A data architecture paradigm that combines the best features of data lakes and data warehouses, providing ACID transactions, schema enforcement, an... |
| Parquet | Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides efficient data co... |