# Godot 4 — Data Pipeline

> **Purpose**: Data-driven architecture for RPGs, card games, and content-heavy projects — Custom Resources as typed data, registry patterns, JSON import with validation, hot-reload, and version migration. The patterns that keep a 500-entry database from becoming unmaintainable spaghetti.

---

## 1. Custom Resource — The Foundation

Resources are Godot's answer to "where do I put my game data." Not Dictionaries. Not JSON. Not autoload scripts full of constants.

### When to Use What

| Storage | Use for | Avoid for |
|---------|---------|-----------|
| **Custom Resource** | Anything with typed fields, editor-exposed, referenced across scenes | Throwaway runtime data, player input |
| **Dictionary** | Runtime-only transient data (combat modifiers, temp buffs) | Anything saved to disk or shared between systems |
| **JSON** | External/modder data, translations, bulk content pipelines | Complex nested typed data (no validation, no autocomplete) |

The real answer: **Resources for design-time data, Dictionaries for runtime-only data, JSON for import pipelines that feed into Resources.**

### Template: Typed Resource with Validation

```gdscript
# character_data.gd
class_name CharacterData
extends Resource

@export var id: StringName = &""
@export var display_name: String = ""
@export var portrait_path: String = ""
@export var dialogue_id: StringName = &""

@export_group("Stats")
@export var max_hp: int = 100
@export var attack: int = 10
@export var defense: int = 5
@export var speed: float = 1.0

@export_group("Tags")
@export var tags: Array[StringName] = []


func _init() -> void:
	# Validation runs on load AND on creation in editor
	if id == &"" and display_name != "":
		push_warning("CharacterData '%s' has no id" % display_name)


func validate() -> Array[String]:
	var errors: Array[String] = []
	if id == &"":
		errors.append("missing id")
	if display_name == "":
		errors.append("missing display_name")
	if max_hp <= 0:
		errors.append("max_hp must be > 0, got %d" % max_hp)
	if portrait_path != "" and not ResourceLoader.exists(portrait_path):
		errors.append("portrait not found: %s" % portrait_path)
	return errors
```

Put `validate()` on every data Resource. Call it at load time, not when the player opens their inventory and gets a null portrait.

### Trap: Shared by Default

Resources are **reference types**. Two nodes loading the same `.tres` share the same instance. Mutate one, you mutate both.

```gdscript
# WRONG — mutates the shared resource, affects every enemy of this type
enemy_data.max_hp -= 10

# RIGHT — duplicate for runtime mutation
var runtime_data := enemy_data.duplicate() as CharacterData
runtime_data.max_hp -= 10
```

`duplicate()` is shallow. If your Resource contains sub-Resources (arrays of other Resources), those inner refs are still shared. Deep copy needs `duplicate(true)` — but that's expensive. Better pattern: keep static data immutable, put mutable state in a separate runtime object.

### Trap: Circular References = Memory Leak

```gdscript
# character_data.gd
@export var nemesis: CharacterData  # Points to another CharacterData

# If nemesis also points back → reference cycle → never freed
```

Resources use reference counting, not garbage collection. Circular refs mean refcount never hits zero. Fix: use IDs, not direct references.

```gdscript
@export var nemesis_id: StringName = &""  # Resolve through Registry at runtime
```

---

## 2. Resource Registry Pattern

An autoload that indexes every data Resource by ID. The single access point for all game data.

```gdscript
# registry.gd — Autoload
class_name Registry
extends Node

# Typed dictionaries: ID → Resource
var characters: Dictionary = {}  # StringName → CharacterData
var items: Dictionary = {}       # StringName → ItemData
var skills: Dictionary = {}      # StringName → SkillData

var _load_errors: Array[String] = []


func _ready() -> void:
	_scan_directory("res://data/characters/", characters, "CharacterData")
	_scan_directory("res://data/items/", items, "ItemData")
	_scan_directory("res://data/skills/", skills, "SkillData")

	if _load_errors.size() > 0:
		push_error("Registry: %d errors during load:\n%s" % [
			_load_errors.size(),
			"\n".join(_load_errors)
		])


func get_character(id: StringName) -> CharacterData:
	if not characters.has(id):
		push_error("Registry: unknown character '%s'" % id)
		return null
	return characters[id]


func get_item(id: StringName) -> ItemData:
	if not items.has(id):
		push_error("Registry: unknown item '%s'" % id)
		return null
	return items[id]


func _scan_directory(path: String, target: Dictionary, expected_class: String) -> void:
	var dir := DirAccess.open(path)
	if dir == null:
		_load_errors.append("Cannot open directory: %s" % path)
		return

	dir.list_dir_begin()
	var file_name := dir.get_next()

	while file_name != "":
		if not dir.current_is_dir() and file_name.ends_with(".tres"):
			var full_path := path.path_join(file_name)
			var res := load(full_path)

			if res == null:
				_load_errors.append("Failed to load: %s" % full_path)
			elif res.get("id") == null or res.id == &"":
				_load_errors.append("No id field in: %s" % full_path)
			elif target.has(res.id):
				_load_errors.append("Duplicate id '%s' in: %s" % [res.id, full_path])
			else:
				# Run validation if the resource supports it
				if res.has_method("validate"):
					var errors: Array[String] = res.validate()
					for err in errors:
						_load_errors.append("%s [%s]: %s" % [full_path, res.id, err])

				target[res.id] = res

		file_name = dir.get_next()

	dir.list_dir_end()
```

### Preload vs Lazy Load

| Strategy | When | Tradeoff |
|----------|------|----------|
| **Preload all** (`_ready`) | < 200 resources, fast game startup | Simple, everything available immediately. Slow startup if data grows |
| **Lazy load** (on first access) | 500+ resources, open world | First access has a hitch. Need null checks everywhere |
| **Area-based batch load** | Large RPGs, chapter-based | Load `res://data/chapter_02/` when entering chapter 2. Best of both worlds |

For most indie games: preload all. You'll hit the "too slow" wall around 500+ resources, and by then you know which areas need lazy loading.

### Trap: Load Order

Registry must be an autoload loaded BEFORE any scene that calls `Registry.get_character()`. In Project Settings > Autoload, order matters — drag Registry above any gameplay autoloads.

If a `_ready()` in your main scene calls `Registry.get_character()` before Registry's own `_ready()` has run, you get null. Autoloads initialize in list order. Scene `_ready()` fires after all autoloads.

---

## 3. JSON Import Pipeline

For content that lives outside `.tres` files — bulk data, designer spreadsheets exported as JSON, modding support.

### Loader with Type Conversion

```gdscript
# data_importer.gd
class_name DataImporter
extends RefCounted

signal progress_updated(current: int, total: int)

var _errors: Array[String] = []


func get_errors() -> Array[String]:
	return _errors


func import_characters(json_path: String) -> Array[CharacterData]:
	_errors.clear()
	var raw := _load_json(json_path)
	if raw == null:
		return []

	if raw is not Array:
		_errors.append("%s: root must be an Array" % json_path)
		return []

	var results: Array[CharacterData] = []
	var total: int = raw.size()

	for i in range(total):
		var entry: Variant = raw[i]
		if entry is not Dictionary:
			_errors.append("%s[%d]: expected object, got %s" % [json_path, i, typeof(entry)])
			continue

		var data := _parse_character(entry, json_path, i)
		if data != null:
			results.append(data)

		if i % 50 == 0:
			progress_updated.emit(i, total)

	progress_updated.emit(total, total)
	return results


func _parse_character(entry: Dictionary, source: String, index: int) -> CharacterData:
	var ctx := "%s[%d]" % [source, index]

	# Required fields
	var id: Variant = entry.get("id")
	if id is not String or id == "":
		_errors.append("%s: missing or empty 'id'" % ctx)
		return null

	var data := CharacterData.new()
	data.id = StringName(id)
	data.display_name = str(entry.get("name", ""))
	data.portrait_path = str(entry.get("portrait", ""))
	data.dialogue_id = StringName(str(entry.get("dialogue_id", "")))

	# JSON numbers are ALWAYS float — cast explicitly
	data.max_hp = int(entry.get("max_hp", 100))
	data.attack = int(entry.get("attack", 10))
	data.defense = int(entry.get("defense", 5))
	data.speed = float(entry.get("speed", 1.0))

	# Tags: array of strings → array of StringName
	var raw_tags: Variant = entry.get("tags", [])
	if raw_tags is Array:
		for tag in raw_tags:
			data.tags.append(StringName(str(tag)))

	return data


func _load_json(path: String) -> Variant:
	if not FileAccess.file_exists(path):
		_errors.append("File not found: %s" % path)
		return null

	var file := FileAccess.open(path, FileAccess.READ)
	var text := file.get_as_text()
	file.close()

	var json := JSON.new()
	var err := json.parse(text)

	if err != OK:
		_errors.append("%s line %d: %s" % [path, json.get_error_line(), json.get_error_message()])
		return null

	return json.data
```

### Trap: JSON Numbers Are Always Float

Godot's JSON parser returns all numbers as `float`. `"max_hp": 100` becomes `100.0`. If you assign that to an `int` variable without casting, it works (implicit conversion) — but comparisons can fail:

```gdscript
var hp: Variant = json_data["max_hp"]  # 100.0 (float)
if hp == 100:    # true — Godot coerces
	pass
if hp is int:    # FALSE — it's a float
	pass
```

Always `int()` explicitly when pulling integers from JSON.

### Trap: Large Files — Batch with Progress

1000+ entries parsed in a single frame = visible hitch. The `progress_updated` signal in the importer lets you drive a loading bar. For truly massive datasets (10k+), split the work across frames:

```gdscript
# In a loading screen coroutine
var importer := DataImporter.new()
importer.progress_updated.connect(_on_progress)

# Run in a thread for zero frame drops
var thread := Thread.new()
thread.start(importer.import_characters.bind("res://data/characters.json"))

# In _process, check thread.is_alive() and join when done
```

---

## 4. Schema Validation

Don't validate at use time. Validate at load time. Fail fast, report everything.

```gdscript
# schema_validator.gd
class_name SchemaValidator
extends RefCounted

enum FieldType { STRING, INT, FLOAT, BOOL, ARRAY, DICT }

var _errors: Array[String] = []


func get_errors() -> Array[String]:
	return _errors


func validate(data: Dictionary, schema: Dictionary, context: String = "") -> bool:
	_errors.clear()
	_validate_dict(data, schema, context)
	return _errors.size() == 0


func _validate_dict(data: Dictionary, schema: Dictionary, ctx: String) -> void:
	# Check required fields
	for field_name: String in schema:
		var spec: Dictionary = schema[field_name]
		var required: bool = spec.get("required", true)

		if not data.has(field_name):
			if required:
				_errors.append("%s: missing required field '%s'" % [ctx, field_name])
			continue

		var value: Variant = data[field_name]
		var expected_type: FieldType = spec.get("type", FieldType.STRING)

		# Type check
		if not _type_matches(value, expected_type):
			_errors.append("%s.%s: expected %s, got %s" % [
				ctx, field_name, FieldType.keys()[expected_type], type_string(typeof(value))
			])
			continue

		# Range check for numbers
		if spec.has("min") and value < spec["min"]:
			_errors.append("%s.%s: value %s below minimum %s" % [ctx, field_name, value, spec["min"]])
		if spec.has("max") and value > spec["max"]:
			_errors.append("%s.%s: value %s above maximum %s" % [ctx, field_name, value, spec["max"]])

		# Enum check (allowed values)
		if spec.has("enum") and value not in spec["enum"]:
			_errors.append("%s.%s: '%s' not in allowed values %s" % [ctx, field_name, value, spec["enum"]])

		# Nested object validation
		if expected_type == FieldType.DICT and spec.has("schema"):
			_validate_dict(value, spec["schema"], "%s.%s" % [ctx, field_name])

		# Array of objects validation
		if expected_type == FieldType.ARRAY and spec.has("item_schema"):
			for i in range(value.size()):
				if value[i] is Dictionary:
					_validate_dict(value[i], spec["item_schema"], "%s.%s[%d]" % [ctx, field_name, i])


func _type_matches(value: Variant, expected: FieldType) -> bool:
	match expected:
		FieldType.STRING: return value is String
		FieldType.INT: return value is int or value is float  # JSON quirk
		FieldType.FLOAT: return value is float or value is int
		FieldType.BOOL: return value is bool
		FieldType.ARRAY: return value is Array
		FieldType.DICT: return value is Dictionary
	return false
```

### Usage: Define Schema as Data

```gdscript
const CHARACTER_SCHEMA: Dictionary = {
	"id": { "type": SchemaValidator.FieldType.STRING, "required": true },
	"name": { "type": SchemaValidator.FieldType.STRING, "required": true },
	"max_hp": { "type": SchemaValidator.FieldType.INT, "required": true, "min": 1, "max": 99999 },
	"attack": { "type": SchemaValidator.FieldType.INT, "required": false, "min": 0 },
	"element": { "type": SchemaValidator.FieldType.STRING, "required": false, "enum": ["fire", "ice", "lightning", "none"] },
	"skills": {
		"type": SchemaValidator.FieldType.ARRAY,
		"required": false,
		"item_schema": {
			"skill_id": { "type": SchemaValidator.FieldType.STRING, "required": true },
			"level": { "type": SchemaValidator.FieldType.INT, "required": true, "min": 1 },
		}
	}
}

# At load time
var validator := SchemaValidator.new()
for i in range(raw_data.size()):
	if not validator.validate(raw_data[i], CHARACTER_SCHEMA, "characters[%d]" % i):
		for err in validator.get_errors():
			push_error(err)
```

Report ALL errors, not just the first. A designer who fixes one field and re-imports only to discover the next error will hate you. Show everything at once.

### Trap: Optional Fields Need Explicit Defaults

```gdscript
# WRONG — optional field with no default, silent null at runtime
data.element = entry.get("element")  # null if missing

# RIGHT — explicit default
data.element = entry.get("element", "none")
```

Every optional field must have a documented default. "Optional" means "has a sane default," not "might be null."

---

## 5. Hot-Reload for Debug

Change a JSON file, see the result in-game without restarting. Critical for designers iterating on balance.

```gdscript
# hot_reloader.gd — attach to Registry or as a debug child
class_name HotReloader
extends Node

signal data_reloaded(source: String)

@export var watch_paths: Array[String] = ["res://data/"]
@export var poll_interval: float = 1.0

var _file_timestamps: Dictionary = {}  # path → modification time
var _timer: float = 0.0
var _reload_queued: bool = false
var _pending_sources: Array[String] = []


func _ready() -> void:
	if not OS.is_debug_build():
		queue_free()
		return
	_scan_timestamps()


func _process(delta: float) -> void:
	_timer += delta
	if _timer < poll_interval:
		return
	_timer = 0.0

	if _reload_queued:
		_execute_reload()
		return

	_check_for_changes()


func _check_for_changes() -> void:
	for base_path in watch_paths:
		_check_directory(base_path)


func _check_directory(path: String) -> void:
	var dir := DirAccess.open(path)
	if dir == null:
		return

	dir.list_dir_begin()
	var file_name := dir.get_next()

	while file_name != "":
		var full_path := path.path_join(file_name)
		if dir.current_is_dir():
			_check_directory(full_path)
		elif file_name.ends_with(".json") or file_name.ends_with(".tres"):
			var mtime := FileAccess.get_modified_time(full_path)
			if _file_timestamps.get(full_path, 0) != mtime:
				_file_timestamps[full_path] = mtime
				_pending_sources.append(full_path)
				_reload_queued = true

		file_name = dir.get_next()
	dir.list_dir_end()


func _execute_reload() -> void:
	_reload_queued = false
	var sources := _pending_sources.duplicate()
	_pending_sources.clear()

	for source in sources:
		print("[HotReload] Reloading: %s" % source)
		data_reloaded.emit(source)


func _scan_timestamps() -> void:
	for base_path in watch_paths:
		_scan_timestamps_recursive(base_path)


func _scan_timestamps_recursive(path: String) -> void:
	var dir := DirAccess.open(path)
	if dir == null:
		return

	dir.list_dir_begin()
	var file_name := dir.get_next()

	while file_name != "":
		var full_path := path.path_join(file_name)
		if dir.current_is_dir():
			_scan_timestamps_recursive(full_path)
		elif file_name.ends_with(".json") or file_name.ends_with(".tres"):
			_file_timestamps[full_path] = FileAccess.get_modified_time(full_path)
		file_name = dir.get_next()
	dir.list_dir_end()
```

### Connecting to Registry

```gdscript
# In Registry._ready():
if OS.is_debug_build():
	var reloader := HotReloader.new()
	reloader.data_reloaded.connect(_on_data_file_changed)
	add_child(reloader)

func _on_data_file_changed(source: String) -> void:
	if "characters" in source:
		_scan_directory("res://data/characters/", characters, "CharacterData")
		print("[Registry] Characters reloaded (%d entries)" % characters.size())
```

### Trap: Reload During Gameplay = Crash

If a combat system holds a reference to `CharacterData` and you hot-reload that resource, the old reference is now stale. The combat reads old stats, or worse, a freed object.

Queue reload for a safe point: between turns, during a pause, on scene transition. Never mid-frame during gameplay logic.

```gdscript
# Safe reload pattern
var _reload_pending: bool = false

func _on_data_file_changed(_source: String) -> void:
	_reload_pending = true  # Don't reload NOW

func on_safe_point() -> void:  # Called by game state machine at transition
	if _reload_pending:
		_reload_pending = false
		_execute_full_reload()
```

---

## 6. Data Migration Between Versions

Players don't update linearly. Someone on v1 save data might update straight to v5. Every migration step must be preserved forever.

```gdscript
# migrator.gd
class_name DataMigrator
extends RefCounted

# Each migration: callable that takes Dictionary, returns Dictionary
var _migrations: Dictionary = {}  # int (from_version) → Callable


func _init() -> void:
	_migrations[1] = _migrate_v1_to_v2
	_migrations[2] = _migrate_v2_to_v3
	_migrations[3] = _migrate_v3_to_v4


func migrate(data: Dictionary, target_version: int) -> Dictionary:
	var current_version: int = data.get("version", 1)

	while current_version < target_version:
		if not _migrations.has(current_version):
			push_error("No migration from v%d to v%d" % [current_version, current_version + 1])
			return data

		print("[Migrator] v%d → v%d" % [current_version, current_version + 1])
		data = _migrations[current_version].call(data)
		current_version += 1
		data["version"] = current_version

	return data


# --- Migration functions (NEVER delete these) ---

func _migrate_v1_to_v2(data: Dictionary) -> Dictionary:
	# v2: renamed "hp" to "max_hp", added "defense" field
	if data.has("hp"):
		data["max_hp"] = data["hp"]
		data.erase("hp")
	if not data.has("defense"):
		data["defense"] = 5  # Sensible default for existing saves
	return data


func _migrate_v2_to_v3(data: Dictionary) -> Dictionary:
	# v3: "element" changed from string to array (multi-element support)
	if data.has("element") and data["element"] is String:
		var old_element: String = data["element"]
		data["elements"] = [old_element] if old_element != "none" else []
		data.erase("element")
	return data


func _migrate_v3_to_v4(data: Dictionary) -> Dictionary:
	# v4: "skills" changed from Array[String] (ids) to Array[Dictionary] (id + level)
	if data.has("skills") and data["skills"].size() > 0 and data["skills"][0] is String:
		var old_skills: Array = data["skills"]
		var new_skills: Array[Dictionary] = []
		for skill_id in old_skills:
			new_skills.append({ "skill_id": skill_id, "level": 1 })
		data["skills"] = new_skills
	return data
```

### Trap: Never Delete Old Migrations

A player who hasn't opened the game since v1 updates to v5. The chain runs: v1→v2→v3→v4→v5. Delete `_migrate_v1_to_v2` and that player's save is bricked. Keep every migration function forever — they're small, they're cheap, they're insurance.

### Trap: Renamed Fields Need Explicit Handling

```gdscript
# WRONG — just adding the new field, ignoring the old one
data["max_hp"] = data.get("max_hp", 100)  # Old "hp" field silently lost

# RIGHT — rename explicitly
if data.has("hp"):
	data["max_hp"] = data["hp"]
	data.erase("hp")
```

The old key doesn't magically become the new key. Every rename needs a migration step that copies the value and erases the old key.

---

## 7. Design Traps

Named traps that kill data-driven projects. Not API bugs — design mistakes.

**Trap: String IDs Without Validation**

`Registry.get_character("npc_01")` — one typo and you get null at runtime, no error at load time. Fix: validate all cross-references at load. If a skill references `"character_id": "npc_01"`, check that ID exists in the character registry during import.

```gdscript
# In your import pipeline, after all registries are loaded:
func validate_cross_references() -> void:
	for skill: SkillData in Registry.skills.values():
		if skill.target_id != &"" and not Registry.characters.has(skill.target_id):
			push_error("Skill '%s' references unknown character '%s'" % [skill.id, skill.target_id])
```

**Trap: File Paths as IDs**

Using `"res://data/characters/npc_01.tres"` as the identifier. Move or rename the file and every reference breaks. IDs must be abstract — `&"npc_01"`, stored inside the resource, independent of file location.

**Trap: No Static/Runtime Separation**

Static data = what the designer authored (base stats, descriptions, skill definitions). Runtime data = what changes during play (current HP, equipped items, cooldowns). These MUST be separate structures.

```gdscript
# Static — loaded from disk, never mutated
var base_data: CharacterData = Registry.get_character(&"hero")

# Runtime — created per-instance, mutated freely
var runtime := CharacterRuntime.new(base_data)
runtime.current_hp -= 20  # Fine, this is the runtime copy
```

Mixing them means either you corrupt your source data, or you try to save runtime state back into design-time resources and lose the original values.

**Trap: Loading Everything at Startup**

200 resources? Preload, no problem. 2000 resources across 8 areas? Your loading screen is now 6 seconds and your memory baseline is 400MB. Batch load by area or chapter. The player exploring Area 1 doesn't need Area 7's enemy data in RAM.

**Trap: Resources Look Like Value Types But Aren't**

```gdscript
var a: CharacterData = Registry.get_character(&"npc_01")
var b: CharacterData = Registry.get_character(&"npc_01")
a.max_hp = 999
print(b.max_hp)  # 999 — same object
```

Every access returns the same instance. If you need a mutable copy, `duplicate()`. If you're never mutating static data (and you shouldn't be), this isn't a problem — but it will bite you the one time someone writes `data.max_hp += buff` without thinking.

---

## Combination Stack

How everything fits together in a real project:

```
res://
├── data/
│   ├── characters/           # .tres files (CharacterData)
│   │   ├── hero.tres
│   │   ├── npc_01.tres
│   │   └── boss_fire.tres
│   ├── items/                # .tres files (ItemData)
│   ├── skills/               # .tres files (SkillData)
│   └── import/               # .json files (external pipeline)
│       ├── characters.json
│       └── translations.json
├── scripts/
│   ├── data/
│   │   ├── character_data.gd     # Custom Resource definition
│   │   ├── item_data.gd
│   │   ├── skill_data.gd
│   │   └── character_runtime.gd  # Runtime mutable wrapper
│   ├── systems/
│   │   ├── registry.gd           # Autoload — preloads all .tres, indexes by ID
│   │   ├── data_importer.gd      # JSON → typed Resources
│   │   ├── schema_validator.gd   # Validates structure at load time
│   │   ├── data_migrator.gd      # Version chain migration
│   │   └── hot_reloader.gd       # Debug-only file watcher
```

### Load Sequence

```
1. Registry._ready()
   ├── Scan res://data/characters/ → load .tres → validate → index by id
   ├── Scan res://data/items/      → same
   ├── Scan res://data/skills/     → same
   ├── validate_cross_references() → check all inter-resource IDs exist
   └── (debug) attach HotReloader → watch res://data/ for changes

2. JSON import (optional, at startup or on demand)
   ├── DataImporter.import_characters("res://data/import/characters.json")
   ├── SchemaValidator.validate() each entry against CHARACTER_SCHEMA
   ├── Convert to CharacterData instances → register in Registry
   └── Report ALL errors at once

3. Gameplay accesses data
   ├── var base := Registry.get_character(&"hero")    # Static, immutable
   ├── var runtime := CharacterRuntime.new(base)       # Mutable per-instance
   └── runtime.take_damage(20)                         # Mutates runtime only

4. Save/Load cycle
   ├── Save: serialize runtime data + version number
   ├── Load: DataMigrator.migrate(data, CURRENT_VERSION)
   └── Reconstruct runtime from migrated data + static base
```

The result: a data pipeline where the designer edits `.tres` files in the Godot inspector (or exports JSON from a spreadsheet), the game validates everything at load time with clear error messages, runtime state never corrupts source data, and saves survive version updates through chained migration. Hot-reload lets balance iteration happen without restarting the game.
