[Bug]: Node.hash uses MetadataMode.ALL, causing unnecessary re-embeds when volatile file-stat metadata changes

[stirelli]

Bug Description

Since PR #18303 (merged 2025-03-30), Node.hash includes metadata via MetadataMode.ALL. That mode ignores excluded_embed_metadata_keys, so every metadata field ends up in the hash. This includes file-stat-derived fields that SimpleDirectoryReader populates via default_file_metadata_func (last_modified_date, creation_date, last_accessed_date, file_size).

Under IngestionPipeline._handle_upserts, a hash mismatch triggers vector_store.delete() followed by a full re-embed. Combined with MetadataMode.ALL, this means any modification to volatile filesystem metadata will trigger a full re-embedding of the document's chunks on the next ingestion run, even when the text content is byte-identical.

The trigger is silent: no warning, no log, no indication that the pipeline is re-embedding unchanged content. The cost scales linearly with corpus size, modification rate, and re-indexing frequency.

Root cause

PR #18303 correctly addressed #17871 (metadata changes not being detected by IngestionPipeline) by adding metadata to the hash. The fix used MetadataMode.ALL rather than MetadataMode.EMBED. Since excluded_embed_metadata_keys exists precisely to mark fields as volatile, operational, or not content-relevant, including them in the hash defeats the purpose of that mechanism.

Scope

The bug manifests under these conditions:

• SimpleDirectoryReader over a local filesystem. last_modified_date is formatted as "%Y-%m-%d" (date only), so the trigger is cross-day modifications. Any file modified between scheduled ingestion runs re-embeds on the next run.
• Any reader that populates temporal metadata with sub-day precision (via a custom file_metadata callable, or via get_resource_info() code paths). Every modification triggers re-embedding, at the format's precision.
• Manually-constructed Documents with volatile metadata. Same as above.

The bug does not manifest for SimpleDirectoryReader over fsspec cloud backends (s3fs, gcsfs, adlfs) in the current default load_data() path. This is because default_file_metadata_func (readers/file/base.py:164-168) queries POSIX-style stat keys (mtime, atime, created) that fsspec backends don't emit. The fix for that separate gap (using fs.modified(path) instead) would simultaneously activate this bug for every fsspec-backed reader.

Reproducers

Five progressive reproducers in this repo: https://github.com/stirelli/llamaindex-embedding-churn

Minimal reproduction (no API calls, no cost):

import os
from datetime import datetime, timedelta
from llama_index.core import Document, SimpleDirectoryReader
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.vector_stores import SimpleVectorStore
from llama_index.core.embeddings import BaseEmbedding
from pydantic import PrivateAttr
from pathlib import Path

class CountingEmbedding(BaseEmbedding):
    _call_count: int = PrivateAttr(default=0)
    @classmethod
    def class_name(cls): return "Counting"
    def _get_text_embedding(self, text):
        self._call_count += 1
        return [0.0] * 1536
    def _get_query_embedding(self, q): return self._get_text_embedding(q)
    async def _aget_text_embedding(self, t): return self._get_text_embedding(t)
    async def _aget_query_embedding(self, q): return self._get_text_embedding(q)
    def _get_text_embeddings(self, ts): return [self._get_text_embedding(t) for t in ts]
    async def _aget_text_embeddings(self, ts): return self._get_text_embeddings(ts)

docs_dir = Path("/tmp/churn_repro")
docs_dir.mkdir(exist_ok=True)
(docs_dir / "doc.md").write_text("Hello, world." * 100)
os.utime(docs_dir / "doc.md", ((datetime.now() - timedelta(days=1)).timestamp(),) * 2)

embedder = CountingEmbedding()
pipeline = IngestionPipeline(
    transformations=[SentenceSplitter(chunk_size=256), embedder],
    docstore=SimpleDocumentStore(),
    vector_store=SimpleVectorStore(),
)

pipeline.run(documents=SimpleDirectoryReader(input_dir=str(docs_dir)).load_data())
print(f"Phase 1 (initial):  embed_calls={embedder._call_count}")

pipeline.run(documents=SimpleDirectoryReader(input_dir=str(docs_dir)).load_data())
print(f"Phase 2 (no change): embed_calls={embedder._call_count}  # cache works")

(docs_dir / "doc.md").touch()  # bump mtime to today
pipeline.run(documents=SimpleDirectoryReader(input_dir=str(docs_dir)).load_data())
print(f"Phase 3 (post-touch): embed_calls={embedder._call_count}  # bug fires")

Expected behavior:

• Phase 1 (initial): some positive number N of embed calls (depends on chunking).
• Phase 2 (no change): same as Phase 1. Cache hit, no new calls.
• Phase 3 (post-touch): embed count has doubled to 2N. This is an unnecessary re-embed of byte-identical content, triggered by a single touch that moved mtime across a calendar day.

Proposed fix

Three options to consider. Each preserves the intent of #18303 (detect meaningful metadata changes) to varying degrees.

Option A (recommended): Use MetadataMode.EMBED instead of MetadataMode.ALL

One line change in Node.hash:

# llama-index-core/llama_index/core/schema.py, Node.hash @property
-        metadata_str = self.get_metadata_str(mode=MetadataMode.ALL)
+        metadata_str = self.get_metadata_str(mode=MetadataMode.EMBED)

This respects excluded_embed_metadata_keys, the mechanism already present in the codebase for exactly this purpose. SimpleDirectoryReader already adds the volatile file-stat fields to excluded_embed_metadata_keys by default:

# llama-index-core/llama_index/core/readers/file/base.py
excluded_embed_metadata_keys=[
    "file_name", "file_type", "file_size",
    "creation_date", "last_modified_date", "last_accessed_date",
],

so the fix correctly excludes volatile metadata from hash computation without requiring any reader-side changes.

Meaningful metadata changes (user-added tags, custom enrichments) remain tracked because excluded_embed_metadata_keys contains only the fields the reader explicitly marked as excluded from embedding. That matches the intent of #17871.

Pros:

• One-line diff.
• Reuses existing infrastructure (no new API surface).
• Preserves the [Bug]: Hash of Node does not consider metadata #17871 fix for content-relevant metadata changes.

Cons:

• Conflates two concerns in excluded_embed_metadata_keys: "don't send to embedder" and "don't invalidate cached embedding." In practice these align (if a field isn't sent to the embedder, the embedding output won't change when the field changes, so re-embedding is waste), but a maintainer may prefer a dedicated mechanism.

Option B: Remove metadata from the hash entirely (content-only hash)

Revert the metadata inclusion added by #18303. The hash becomes a pure function of the text content.

Pros:

• Cleanest semantics. No ambiguity about what triggers re-embed.
• Smallest possible surface for the hash.

Cons:

• Regresses [Bug]: Hash of Node does not consider metadata #17871. Users who rely on metadata updates triggering re-embed (the original bug Include Node metadata in hash calculation #18303 was designed to fix) lose that behavior.
• Shifts the burden back to users to explicitly invalidate documents when metadata changes.

Option C: Add a dedicated excluded_hash_metadata_keys field

Introduce a new attribute on Node / Document that controls which metadata fields participate in the hash, independent of excluded_embed_metadata_keys.

Pros:

• Cleanest separation of concerns. Embedding exclusion and hash exclusion are different user-facing levers.
• Supports edge cases (e.g., a field excluded from embedding but wanted in the hash, or vice versa).

Cons:

• API surface growth. Every reader that populates volatile metadata would need to populate the new field too.
• More breaking for downstream users who have already built around the current two-field model.
• Defaults have to be chosen carefully (should it default to excluded_embed_metadata_keys? to empty? something else?).

My recommendation

Option A. It fixes the bug with a one-line change, preserves the #17871 fix, and uses existing infrastructure. Options B and C are legitimate but come with larger trade-offs. Happy to be overruled by whichever option the maintainers prefer.

Impact

• Production deployments using scheduled SimpleDirectoryReader ingestion over local corpora that change between runs re-embed modified files on every cross-day ingestion cycle, silently.
• Users adding custom file_metadata callables that include timestamps (a natural pattern) experience re-embedding on every source modification at the format's precision.
• get_resource_info() code paths in S3Reader, GCSReader, and similar readers populate sub-second datetime fields and would fire this bug on every source update.
• Users who manually construct Documents with any volatile metadata field experience the same behavior.

Version

llama-index-core==0.14.21 (current at time of writing). Bug introduced in #18303, merged into llama-index-core prior to v0.12.20 release.

Steps to reproduce

See reproducer above, or run the repo at https://github.com/stirelli/llamaindex-embedding-churn. It has five progressive levels of evidence, including end-to-end verification against a real OpenAI API key for billed-cost confirmation.

I'd be happy to

Open a PR with the one-line fix plus a unit test demonstrating the regression is prevented, if the maintainers agree with the approach. Alternative fixes, such as adding a hash_mode parameter to give users control, are also possible. Happy to discuss the trade-off here first.

[stirelli]

Update: during preparation of the PR I discovered the fix in Node.hash alone is insufficient. The bug manifests through three interacting hash mechanisms, all of which use MetadataMode.ALL (or equivalent unfiltered metadata inclusion):

1. Node.hash at llama-index-core/llama_index/core/schema.py:749 (the one this issue originally described). Used for Document upsert comparison.
2. TextNode.hash at llama-index-core/llama_index/core/schema.py:801. Uses str(self.metadata) directly. This is the active path for nodes produced by SentenceSplitter and other splitters, so it's what IngestionPipeline._handle_upserts actually compares in practice.
3. get_transformation_hash at llama-index-core/llama_index/core/ingestion/pipeline.py:62. Uses node.get_content(metadata_mode=MetadataMode.ALL). This is the IngestionCache key for transformation output, so even if node-level hashes are fixed, the transformation cache still misses on volatile metadata change and re-runs the whole transformation chain (including the embedder).

All three need to change to MetadataMode.EMBED (or the equivalent exclusion-respecting filter) for the bug to be fully fixed. The upcoming PR applies all three changes together with the regression test.

This also answers the question I hedged on in the original post: "is this really the only code path?" It isn't. The three places are parallel implementations of the same concept (hash that includes metadata) that were added or modified at different points in the codebase's history without being unified.