Skip to content

No polars in parquet loaders - #2282

Merged
ljeub-pometry merged 20 commits into
docbrownv4_mergefrom
no_polars_in_parquet
Sep 17, 2025
Merged

No polars in parquet loaders#2282
ljeub-pometry merged 20 commits into
docbrownv4_mergefrom
no_polars_in_parquet

Conversation

@ljeub-pometry

@ljeub-pometry ljeub-pometry commented Sep 16, 2025

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

Replace polars-arrow with arrow-rs for parquet and data frame loaders

Why are the changes needed?

Does this PR introduce any user-facing change? If yes is this documented?

How was this patch tested?

Are there any further changes required?

wyatt-joyner-pometry and others added 19 commits August 12, 2025 21:33
@wyatt-joyner-pometry @github-actions
Fix/top k (#2228)
f5e9fae 
* add linear top_k implementation

* update top_k to use new linear top_k implementation

* cleanup

* chore: apply tidy-public auto-fixes

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@wyatt-joyner-pometry @github-actions
Fix/fastrp (#2229)
aadb0bd 
* update fastrp to properly average embeddings instead of just summing

* fix import issue

* add helper function to test pairwise distances

* cleanup and update python test

* chore: apply tidy-public auto-fixes

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@ricopinazo @github-actions
fix docker ci (#2227)
82f785f 
* sort ci action and create new Dockerfile for python

* add missing dockerfile

* remove env from docker ci action

* fix python dockerfile

* chore: apply tidy-public auto-fixes

* add action input to be able to build the python image

* add permissions

* read base input

* fix typo

* change default branch to master

* chore: apply tidy-public auto-fixes

* fix using wrong username on merge step

* change manual docker release to also build python

* fix digest name conflicts

* change choice to boolean for action input

* fix digest name using old variable

* sort versioning and add nightly action

* add permissions to nightly action

* add permissions to manual docker release action

* read dry-run from the docker action

* add dry_run to docker action

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@ricopinazo @github-actions
graphql bench on CI and vector bench (#2198)
ba0ca3d 
* rewrite the vector bench script

* try disabling available memory setting

* try arroy append api

* add graphql benchmarks

* chore: apply tidy-public auto-fixes

* add parallel workflow for graphql bench

* chore: apply tidy-public auto-fixes

* setup python

* setup k6

* chore: apply tidy-public auto-fixes

* get output file out of the results folder

* add pnpm-workspace.yaml

* fix ci error

* chore: apply tidy-public auto-fixes

* fix ci for good this time?

* test gh pages branch

* chore: apply tidy-public auto-fixes

* test different dir for graphql

* reduce target

* add -100 for testing

* chore: apply tidy-public auto-fixes

* add TODO

* chore: apply tidy-public auto-fixes

* cleanup PR

* re-enable base benches

* fix flaky bench

* remove -100

* remove cell from results.ipynb

* sort parent workflow

* add missing bench.ts

* chore: apply tidy-public auto-fixes

* some final bits

* print json output on ci to make sure numbers are valid

* fix ci

* set iteration target to 6400

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@jbaross-pometry @github-actions @miratepuffin
James/graphql docstrings (#2210)
ceafe28 
* init

* docstrings

* docstrings for edges

* docstrings for edges

* regen schema and docs

* run formatting

* chore: apply tidy-public auto-fixes

* more docstrings

* backticks are not used by docs parser so remove

* more docstrings

* chore: apply tidy-public auto-fixes

* more docstrings

* chore: apply tidy-public auto-fixes

* more docstrings

* more docstrings

* update schema and format

* chore: apply tidy-public auto-fixes

* more docstrings

* chore: apply tidy-public auto-fixes

* more docstrings

* chore: apply tidy-public auto-fixes

* more docstrings

* chore: apply tidy-public auto-fixes

* more docstrings

* more docstrings

* cleanup

* more docstrings

* chore: apply tidy-public auto-fixes

* testcase for inputs

* cleanup

* chore: apply tidy-public auto-fixes

* more docstrings

* cleanup

* chore: apply tidy-public auto-fixes

* more docstrings

* cleanup

* chore: apply tidy-public auto-fixes

* fix page  docstrings

* cleanup

* remove latin

* chore: apply tidy-public auto-fixes

* initial fixes

* fmt

* chore: apply tidy-public auto-fixes

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ben Steer <ben.steer@pometry.com>
@github-actions @miratepuffin
Release v0.16.1 (#2236)
b81c0c1 
chore: Release

Co-authored-by: Pometry-Team <ben.steer@pometry.com>
@ljeub-pometry @github-actions
Fix explode layers for filtered persistent graph (#2241)
3e12628 
* explode layers for valid graph is broken

* explode_layers was ignoring layer filters for persistent semantics

* chore: apply tidy-public auto-fixes

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@jbaross-pometry @github-actions @miratepuffin
James/graphql docstrings fixes (#2239)
0a60409 
* init

* docstrings

* docstrings for edges

* docstrings for edges

* regen schema and docs

* run formatting

* chore: apply tidy-public auto-fixes

* more docstrings

* backticks are not used by docs parser so remove

* more docstrings

* chore: apply tidy-public auto-fixes

* more docstrings

* chore: apply tidy-public auto-fixes

* more docstrings

* more docstrings

* update schema and format

* chore: apply tidy-public auto-fixes

* more docstrings

* chore: apply tidy-public auto-fixes

* more docstrings

* chore: apply tidy-public auto-fixes

* more docstrings

* chore: apply tidy-public auto-fixes

* more docstrings

* more docstrings

* cleanup

* more docstrings

* chore: apply tidy-public auto-fixes

* testcase for inputs

* cleanup

* chore: apply tidy-public auto-fixes

* more docstrings

* cleanup

* chore: apply tidy-public auto-fixes

* more docstrings

* cleanup

* chore: apply tidy-public auto-fixes

* fix page  docstrings

* cleanup

* remove latin

* chore: apply tidy-public auto-fixes

* initial fixes

* fmt

* chore: apply tidy-public auto-fixes

* fix double spacing

* review fixes

* specify layers for has_edge

* tidy

* chore: apply tidy-public auto-fixes

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ben Steer <ben.steer@pometry.com>
Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk>
@jbaross-pometry @github-actions @miratepuffin
James/graphql-userguide-16-x (#2233)
d085e74 
* update ui  image

* mutation and views

* persistent and event distinction

* clean up running steps and add cli

* subtitle

* proper hierarchy

* props and metadata examples

* Add troubleshooting

* add missing cli parameter

* chore: apply tidy-public auto-fixes

* add default save location to troubleshooting

* markdown formatting

* chore: apply tidy-public auto-fixes

* chore: apply tidy-public auto-fixes

* Clarify docker basics

* Clarify docker basics

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk>
@ricopinazo @github-actions
fix nightly release action (#2244)
46c3f06 
* fix nightly release action

* fix versions missing

* chore: apply tidy-public auto-fixes

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@ricopinazo @github-actions
add docker retag action (#2245)
5086c5a 
* add docker retag action

* add permissions

* chore: apply tidy-public auto-fixes

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@edsherrington
update Slack invite link (#2252)
f14de7d
@ljeub-pometry
test strings in df loaders
b750577
@ricopinazo
Increase sleep time on graphql bench (#2278)
432df6c 
Update Makefile
@ljeub-pometry
remove polars from parquet and df loading
9eb285f
@dependabot @github-actions @miratepuffin
Bump tracing-subscriber from 0.3.19 to 0.3.20 in the cargo group acro…
4eb2b5a 
…ss 1 directory (#2251)

* Bump tracing-subscriber in the cargo group across 1 directory

Bumps the cargo group with 1 update in the / directory: [tracing-subscriber](https://github.com/tokio-rs/tracing).


Updates `tracing-subscriber` from 0.3.19 to 0.3.20
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](tokio-rs/tracing@tracing-subscriber-0.3.19...tracing-subscriber-0.3.20)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-version: 0.3.20
  dependency-type: direct:production
  dependency-group: cargo
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: apply tidy-public auto-fixes

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk>
@jbaross-pometry @github-actions
community detection (#2276)
b3d9811 
Covers changes for ticket #2268 and adds an introduction to community detection with a tutorial using our existing algorithms and UI. This should apear first in the search results.

* initial introduction

* add karate example

* add karate example

* add assets and tests

* fix tests

* fix tests

* fix tests

* chore: apply tidy-public auto-fixes

* review: simplify node type assignment

* chore: apply tidy-public auto-fixes

* review: swap to csv file

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@jbaross-pometry @github-actions
Use raphtory from python dir (#2275)
e3c8399 
* Use raphtory from python dir

* comment to match

* chore: apply tidy-public auto-fixes

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@ljeub-pometry
add batch_size argument for the low-level functions
1529436
@CLAassistant

CLAassistant commented Sep 17, 2025 •
edited
Loading

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
5 out of 6 committers have signed the CLA.

✅ wyatt-joyner-pometry
✅ jbaross-pometry
✅ ricopinazo
✅ edsherrington
✅ ljeub-pometry
❌ dependabot[bot]
You have signed the CLA already but the status is still pending? Let us recheck it.

@ljeub-pometry
ljeub-pometry merged commit d208e82 into docbrownv4_merge Sep 17, 2025
1 of 2 checks passed
@ljeub-pometry
ljeub-pometry deleted the no_polars_in_parquet branch September 17, 2025 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fabianmurariu fabianmurariu Awaiting requested review from fabianmurariu

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ljeub-pometry @CLAassistant @wyatt-joyner-pometry @ricopinazo @jbaross-pometry @edsherrington