Knowledge Graph Generator - Full Documentation

# Knowledge Graph Generator – Full Process Documentation

## 1. Prepare a Text

Begin with a source text that will be used to generate a Knowledge Graph (KG).

## 2. Initialize a KnowledgeGraphGenerator Object

    kggen = kg.KnowledgeGraphGenerator(
        model="openai/gpt-4o",
        api_key=os.getenv("OPENAI_API_KEY")
    )

Steps 3–5 generate a KG tailored to the specific text. While these graphs remain faithful to the source, they are difficult to compare across texts since entities and relations are context-specific.

## 3. Initialize a New Graph

- If a text is provided, the function resets internal variables for new graph generation.
- If a JSON path is provided, it loads an existing graph:

    kggen.init_graph(path=f"../../Ramban/json/{file_name}_c{chunk_size}.json")

## 4. Extract Entities

Uses an LLM to extract entities from the text. This function automatically calls `init_graph`, so Step 3 is not required if you call this function directly.

You need to load the text file into a variable and provide it to the function.

The `chunk_size` parameter determines how the text is analyzed:

- `chunk_size=0`: process the text as a single chunk (may dilute context).
- `chunk_size>0`: split into multiple chunks, generate subgraphs, and merge them.

    kggen.extract_entities(text=text, chunk_size=chunk_size)

See Appendix A: Context Dilution for details.

## 5. Extract Relations

Uses an LLM to extract relations between the discovered entities.

    kggen.extract_relations()

## 6. Converting Entities to Concepts

Step 4 may produce duplicate entities with the same meaning (due to orthographic, morphological, or synonymic variation). This step uses an LLM to merge them into concepts, where each concept has:

- A canonical name
- Several alternate names

## 7. Relation Normalization

Step 5 may produce multiple relations with the same meaning. The package supports several approaches:

### 7.1 Using LLM to Merge Relations into Predicates

The LLM groups similar relations, assigns each cluster a canonical predicate, and lists alternative labels.

- Advantage: Increases internal similarity within a text graph.
- Pitfall: Different texts may generate different predicates, complicating comparison.

    kggen.relations2ontology(["LLM"])

### 7.2 Align Relations to an Ontology

Provide a known ontology (full or partial). The system compares embeddings of detected relations to ontology relations and selects the closest match.

- Advantage: Produces unified terminology for easier graph comparison.
- Pitfall: Reduces granularity and may incorrectly assign some relations.

    kggen.relations2ontology(["SKOS"])

### 7.3 LLM Validation of Step 7.2

(Planned): Use an LLM to confirm or correct embedding-based ontology mappings.

### 7.4 Multiple Ontologies

Provide multiple ontologies to overcome limitations of a single one. Each relation is mapped to the best matching relation across ontologies.

    kggen.relations2ontology(["CIDOC_CRM", "SKOS", "DUBLIN_CORE"])

## 8. Visualization

Convert the normalized KG into a specific ontology and export to HTML for visualization:

    ontology = "SKOS"  # Use "MIX" to visualize multiple ontologies
    kggen.graph2Ontology(ontology)
    viz = kggen.visualize(f"../../Ramban/vis/{file_name}_c{chunk_size}_ontology_{ontology}.html")

## Appendix A: Context Dilution in Long Texts

When an LLM processes a long passage, attention is divided across many topics, leading to:

- Merging related concepts into broader categories.
- Omission of fine-grained entities in favor of higher-level abstractions.
- Loss of local detail.

Shorter segments yield finer-grained concepts, but less global coherence.

## Appendix B: Normalizing

### Entities → Concepts

Entities are grouped into higher-level concepts. See `entities2concepts` for more details.

### Relations

Extracted predicates may vary in form:

    "is placed on"
    "is part of"
    "contains"
    "dwells on"
    "is above"
    "is in"

### Common Issues

- Variability in phrasing
- Hebrew/English mix
- Morphological or tense variation
- Synonyms
- Directional ambiguity

### Normalization Strategy

1. Define a canonical predicate vocabulary, aligned to SKOS or custom properties:
   - `skos:broader / skos:narrower` → hierarchical
   - `skos:related` → associative
   - Custom properties (e.g., `ex:contains`, `ex:isIn`, `ex:dwellsOn`)
2. Group variants into canonical predicates:
   - Canonical: `contains` → Variants: is in, is placed in
   - Canonical: `isPartOf` → Variants: is part of, belongs to
   - Canonical: `isAbove` → Variants: is over, stands upon

### Methods

- LLM grouping with granularity (LOW, MEDIUM, HIGH).
- Embedding-based clustering, followed by LLM-generated canonical naming.

## Reference Ontologies

### Dublin Core

    dcterms:isPartOf
    dcterms:hasPart
    dcterms:isVersionOf
    dcterms:hasVersion
    dcterms:isFormatOf
    dcterms:hasFormat
    dcterms:references
    dcterms:isReferencedBy
    dcterms:relation

### SKOS

    skos:broader
    skos:narrower
    skos:related
    skos:exactMatch
    skos:closeMatch
    skos:broadMatch
    skos:narrowMatch
    skos:relatedMatch

### CIDOC CRM (selected)

    P1_is_identified_by
    P2_has_type
    P3_has_note
    P4_has_time_span
    P5_consists_of
    P7_took_place_at
    P8_took_place_on_or_within
    P10_falls_within
    P11_had_participant
    P12_occurred_in_the_presence_of
    P13_destroyed
    P14_carried_out_by
    P15_was_influenced_by
    P16_used_specific_object
    P17_was_motivated_by
    P19_was_intended_use_of
    P20_had_specific_purpose
    P21_had_general_purpose
    P25_moved
    P26_moved_to
    P27_moved_from
    P31_has_modified
    P35_has_identified
    P37_assigned
    P38_deassigned
    P39_measured
    P40_observed_dimension
    P43_has_dimension
    P90_has_value
    P94_has_created
    P96_by_mother
    P97_from_father
    P98_brought_into_life
    P100_was_death_of
    P102_has_title
    P104_is_subject_to
    P105_right_held_by
    P127_has_broader_term

## Ontology Mapping Process

The system maps relations into existing ontologies:

- Takes any ontology embedding file (SKOS, Dublin Core, CIDOC CRM, or custom).
- Embeds raw relations from the graph.
- Computes cosine similarity between relation embeddings and ontology embeddings.
- Assigns each raw relation to its closest ontology relation.
- Builds structured `Predicate` objects with:
  - prefLabel_en: ontology relation label (e.g., `skos:broader`, `dcterms:isPartOf`)
  - altLabels_en: list of raw variants mapped to this label
  - mapping_examples: examples of input relations assigned here