Index harness - #2750
Open
miratepuffin wants to merge 14 commits into
Open
Conversation
Storage backends can now accelerate node property filters: - PersistenceStrategy gains defaulted node_prop_candidates / build_node_prop_index hooks, with PropPredicate, PropSemantics and GlobalPropCandidates describing the filter shapes they can serve. - NodePropertyFilterOp pushes eligible filters down to the hooks and narrows its domain to the returned candidates; windowed or layer-restricted views request Ever semantics with exactness off so candidates stay a verified superset. - Index::Sorted holds candidate lists without hashing, and trusted_node_list() lets exact candidate sets skip re-verification. - build_property_index() is exposed on Graph/PersistentGraph in Python and as buildPropertyIndex in GraphQL. Backends without index support are unaffected: the default hooks return None and every filter keeps scanning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FilterOps::list_trusted is the single trust computation behind node_list_trusted, trusted_node_list and Nodes::is_list_filtered. The parquet encoders read the node list and its trust flag through trusted_node_list, and the duplicate Partial/Sorted encoder arms are merged into one. Index::Sorted::union merges through itertools instead of a hand loop. apply_graph_fixture applies a fixture to an existing graph with an id offset; build_graph delegates to it. Docstrings for build_property_index describe behaviour in backend-neutral terms.
| fn pushdown_semantics(&self) -> Option<(NodePropSemantics, bool)> { | ||
| use crate::db::api::view::internal::InnerFilterOps; | ||
| use crate::db::graph::views::filter::model::property_filter::Op; | ||
| let plain_view = !self.graph.window_filtered() && !self.graph.is_layer_filtered(); |
Collaborator
There was a problem hiding this comment.
We probably need per-layer index for a lot of queries? Also can handle Metadata separately as it is not affected by windowing and always has latest semantics?
| fn domain(&self, storage: &GraphStorage) -> NodeList { | ||
| if let Some(candidates) = self.index_candidates(storage) { | ||
| // index candidates are ascending and deduplicated, as `from_sorted` requires | ||
| let list = NodeList::List { |
Collaborator
There was a problem hiding this comment.
This materialises the list multiple times, any way to avoid this?
| .filter(|k| b.contains(k)) | ||
| .collect::<Vec<_>>() | ||
| .into(), | ||
| exact: false, |
Collaborator
There was a problem hiding this comment.
this might be pessimistic for a subgraph, I think we can keep exact on an intersection
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: f59b721 | Previous: 9823ef7 | Ratio |
|---|---|---|---|
large/1k random edge additions with numeric string input |
1797506 ns/iter (± 233416) |
876236 ns/iter (± 86855) |
2.05 |
lotr_graph/num_edges |
5 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
4 ns/iter (± 0) |
1 ns/iter (± 0) |
4 |
lotr_graph/graph_latest |
2 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
6673272 ns/iter (± 32064) |
1564816 ns/iter (± 35303) |
4.26 |
lotr_graph_window_100/num_nodes |
13 ns/iter (± 0) |
5 ns/iter (± 0) |
2.60 |
lotr_graph_window_100_materialise/materialize |
6758270 ns/iter (± 45691) |
1669150 ns/iter (± 10700) |
4.05 |
lotr_graph_window_10_materialise/materialize |
2727895 ns/iter (± 22335) |
971980 ns/iter (± 4278) |
2.81 |
lotr_graph_subgraph_10pc/num_nodes |
14 ns/iter (± 0) |
4 ns/iter (± 0) |
3.50 |
lotr_graph_subgraph_10pc_materialise/materialize |
1870179 ns/iter (± 5693) |
334634 ns/iter (± 1287) |
5.59 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1097557 ns/iter (± 7518) |
230399 ns/iter (± 2617) |
4.76 |
lotr_graph_window_50_layered/has_node_existing |
321 ns/iter (± 25) |
129 ns/iter (± 12) |
2.49 |
lotr_graph_window_50_layered/graph_latest |
77818 ns/iter (± 3812) |
36649 ns/iter (± 916) |
2.12 |
lotr_graph_window_50_layered_materialise/materialize |
24517360 ns/iter (± 150925) |
3488825 ns/iter (± 24948) |
7.03 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
476130 ns/iter (± 4344) |
192686 ns/iter (± 1569) |
2.47 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
41456276 ns/iter (± 130361) |
5298035 ns/iter (± 147912) |
7.82 |
This comment was automatically generated by workflow using github-action-benchmark.
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Shivam <4599890+shivamka1@users.noreply.github.com> # Conflicts: # raphtory/src/db/graph/nodes.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change? If yes is this documented?
How was this patch tested?
Are there any further changes required?