Fix/top k - #2228
Merged
Merged
Conversation
miratepuffin
approved these changes
Aug 12, 2025
ljeub-pometry
added a commit
that referenced
this pull request
Sep 17, 2025
* Fix/top k (#2228) * add linear top_k implementation * update top_k to use new linear top_k implementation * cleanup * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Fix/fastrp (#2229) * update fastrp to properly average embeddings instead of just summing * fix import issue * add helper function to test pairwise distances * cleanup and update python test * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix docker ci (#2227) * sort ci action and create new Dockerfile for python * add missing dockerfile * remove env from docker ci action * fix python dockerfile * chore: apply tidy-public auto-fixes * add action input to be able to build the python image * add permissions * read base input * fix typo * change default branch to master * chore: apply tidy-public auto-fixes * fix using wrong username on merge step * change manual docker release to also build python * fix digest name conflicts * change choice to boolean for action input * fix digest name using old variable * sort versioning and add nightly action * add permissions to nightly action * add permissions to manual docker release action * read dry-run from the docker action * add dry_run to docker action --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * graphql bench on CI and vector bench (#2198) * rewrite the vector bench script * try disabling available memory setting * try arroy append api * add graphql benchmarks * chore: apply tidy-public auto-fixes * add parallel workflow for graphql bench * chore: apply tidy-public auto-fixes * setup python * setup k6 * chore: apply tidy-public auto-fixes * get output file out of the results folder * add pnpm-workspace.yaml * fix ci error * chore: apply tidy-public auto-fixes * fix ci for good this time? * test gh pages branch * chore: apply tidy-public auto-fixes * test different dir for graphql * reduce target * add -100 for testing * chore: apply tidy-public auto-fixes * add TODO * chore: apply tidy-public auto-fixes * cleanup PR * re-enable base benches * fix flaky bench * remove -100 * remove cell from results.ipynb * sort parent workflow * add missing bench.ts * chore: apply tidy-public auto-fixes * some final bits * print json output on ci to make sure numbers are valid * fix ci * set iteration target to 6400 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * James/graphql docstrings (#2210) * init * docstrings * docstrings for edges * docstrings for edges * regen schema and docs * run formatting * chore: apply tidy-public auto-fixes * more docstrings * backticks are not used by docs parser so remove * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * more docstrings * update schema and format * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * more docstrings * cleanup * more docstrings * chore: apply tidy-public auto-fixes * testcase for inputs * cleanup * chore: apply tidy-public auto-fixes * more docstrings * cleanup * chore: apply tidy-public auto-fixes * more docstrings * cleanup * chore: apply tidy-public auto-fixes * fix page docstrings * cleanup * remove latin * chore: apply tidy-public auto-fixes * initial fixes * fmt * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Steer <ben.steer@pometry.com> * Release v0.16.1 (#2236) chore: Release Co-authored-by: Pometry-Team <ben.steer@pometry.com> * Fix explode layers for filtered persistent graph (#2241) * explode layers for valid graph is broken * explode_layers was ignoring layer filters for persistent semantics * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * James/graphql docstrings fixes (#2239) * init * docstrings * docstrings for edges * docstrings for edges * regen schema and docs * run formatting * chore: apply tidy-public auto-fixes * more docstrings * backticks are not used by docs parser so remove * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * more docstrings * update schema and format * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * more docstrings * cleanup * more docstrings * chore: apply tidy-public auto-fixes * testcase for inputs * cleanup * chore: apply tidy-public auto-fixes * more docstrings * cleanup * chore: apply tidy-public auto-fixes * more docstrings * cleanup * chore: apply tidy-public auto-fixes * fix page docstrings * cleanup * remove latin * chore: apply tidy-public auto-fixes * initial fixes * fmt * chore: apply tidy-public auto-fixes * fix double spacing * review fixes * specify layers for has_edge * tidy * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Steer <ben.steer@pometry.com> Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk> * James/graphql-userguide-16-x (#2233) * update ui image * mutation and views * persistent and event distinction * clean up running steps and add cli * subtitle * proper hierarchy * props and metadata examples * Add troubleshooting * add missing cli parameter * chore: apply tidy-public auto-fixes * add default save location to troubleshooting * markdown formatting * chore: apply tidy-public auto-fixes * chore: apply tidy-public auto-fixes * Clarify docker basics * Clarify docker basics --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk> * fix nightly release action (#2244) * fix nightly release action * fix versions missing * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * add docker retag action (#2245) * add docker retag action * add permissions * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * update Slack invite link (#2252) * test strings in df loaders * Increase sleep time on graphql bench (#2278) Update Makefile * remove polars from parquet and df loading * Bump tracing-subscriber from 0.3.19 to 0.3.20 in the cargo group across 1 directory (#2251) * Bump tracing-subscriber in the cargo group across 1 directory Bumps the cargo group with 1 update in the / directory: [tracing-subscriber](https://github.com/tokio-rs/tracing). Updates `tracing-subscriber` from 0.3.19 to 0.3.20 - [Release notes](https://github.com/tokio-rs/tracing/releases) - [Commits](tokio-rs/tracing@tracing-subscriber-0.3.19...tracing-subscriber-0.3.20) --- updated-dependencies: - dependency-name: tracing-subscriber dependency-version: 0.3.20 dependency-type: direct:production dependency-group: cargo ... Signed-off-by: dependabot[bot] <support@github.com> * chore: apply tidy-public auto-fixes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk> * community detection (#2276) Covers changes for ticket #2268 and adds an introduction to community detection with a tutorial using our existing algorithms and UI. This should apear first in the search results. * initial introduction * add karate example * add karate example * add assets and tests * fix tests * fix tests * fix tests * chore: apply tidy-public auto-fixes * review: simplify node type assignment * chore: apply tidy-public auto-fixes * review: swap to csv file --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Use raphtory from python dir (#2275) * Use raphtory from python dir * comment to match * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * add batch_size argument for the low-level functions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: wyatt-joyner-pometry <wyatt.joyner@pometry.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Pedro Rico Pinazo <ricopinazo@gmail.com> Co-authored-by: James Baross <james.baross@pometry.com> Co-authored-by: Ben Steer <ben.steer@pometry.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk> Co-authored-by: edsherrington <ed.sherrington@pometry.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Changing par_top_k to linear top_k and changing the comparison when adding elements to be strict rather than loose.
Why are the changes needed?
This is 50x faster (testing up to 1,000,000,000 elements) than the previous parallel version.
Does this PR introduce any user-facing change? If yes is this documented?
Not in the NodeState API, but internally I have added a top_k method in node_state_ord_ops.
How was this patch tested?
It was tested on the same toy example as par_top_k and evaluated on much larger examples against the output of par_top_k.
Are there any further changes required?
For large enough inputs, we should do a k-merge of N linearly calculated top-k binary heaps. We can also perform some summary stat calculations like avg, min, max, and some bucketing to reduce the search space, but this performs the operation of finding the top 100 elements of 1 billion elements in 14 seconds, so these changes are fairly low priority at the moment.