Add fastwarc-grpc, a streaming gRPC server for FastWARC - #58
Conversation
|
The parser seems to be extremely slow. When I run the benchmark on an uncompressed WARC, I get around 300MiB/s, which is slower than most of the compressed options. Gzip and Zstd are around 480MiB/s, LZ4 is 510MiB/s. For comparison, here's plain FastWARC: My guess is that the streaming mechanism is quite inefficient or that the serving thread doesn't buffer enough so that the consumer is being starved. |
Wild flamegraph. Lot's of kernel TCP IO calls here. FastWARC itself completely disappears. For comparison: This is FastWARC alone. |
|
I'll look into this - there's a lot of things to do to speed this up - I started as minimalist here at first. Standalone pure rust will always be fastest but that flamegraph isn't normal. What tests are you using? I'll use the same ones and get it to speed up. |
|
That is just the profiling binary run with cargo run. I'm using a test WARC file from the CommonCrawl, 5.3GB uncompressed size, 1.2GB compressed (Gzip). |
|
Give you give me the link to the specific file? I just wanna make sure I'm doing apples to apples. gRPC has a lot of tuning - window size is usually the first one that unblocks (the default is 64K and it should be like 50MB for this kind of crawl) |
|
I don't think the exact file matters. I pulled it from our internal storage, so I don't have the upstream link at hand. The file name is |
|
The file won't matter, but if I use the same one it's one less variable I'll question. At first glance it is using the 64kb window - for reference on my own fast machine when it's at 64KB, I can't get more than 3MB/s.. when I set it to 300MB, I can get 1.5GB per second over the wire... so I'm sure I can tune this just a little and can show you better results. |
|
I temporarily put the file here: XXX |
|
Got it .. so for gzip we are getting about 1384MB/s vs. 1430MB/s. ... for uncompressed initially I"m seeing grpc do 4367 MB/s... but that's against in-process uncompressed doing 14179 MB/s on the same machine uncompressed. So that means about a 3x slower service if done as an over-the-wire service. I'm working to make it faster, there's plenty of room for speeding this up - this was just the window tuning so far. I'll get back to ya in a few hours on this. But 100% keeping as draft. I need a couple rounds of tuning. If you don't mind, I'll tune it 1x today and ask you to re-test. You should see a huge improvement, but my gut is saying we'll want 1 more round of testing after that. |
|
Don't run this on a converged SoC machine. It will inflate the numbers. When I run the same program on an M1 MacBook, I get uncompressed 890.4 MiB/s. That's still less than half of the 2698.7 MiB/s for plain FastWARC, though not as bad as the 300MiB/s I get on the other machine with a regular SSD and AMD64 CPU. |
|
FWIW This is the flamegraph on the MacBook. It looks quite different (I mean the shape obviously, different platform, different names), though the actual FastWARC routines are still only about 1/8 of the total time and have a rather flat call stack compared to the async/grpc runtime. And that's just the fastwarc-grpc part. The core FastWARC frames aren't even visible. |
|
Yea, I'm messing with chunk sizes over the wire, getting about the 4.8GB/s localhost and over a 10Gb connection we are getting 1GB/s over the wire - which honestly isn't bad. The idea with the grpc stuff isn't raw speed locally - it's that you can make this a server too. In my case I'm going to use it to take a library of warc files and concurrently run it through a pipeline. I'd have to otherwise do that serialization locally. I'm still doing some tests though. In-process you won't get a faster result since gRPC is made for remote calls. So as we move further from the metal, each layer you see a sharp decline. That being said, I thought to myself "if I take the library and just push the output via UDS on the API - that would be a REAL apples to apples.." .. .but that failed for me as well. Streaming the raw reply make it go from 12.6k MBps to 9.1 MBps - a sizable loss but not nearly as bad as gRPC. So then I realized - the API isn't getting data streaming to it - it is merely getting data already on disk. What if gRPC was in the same playing field, and the server is outputting the results from data on disk: So the real tax here is the serialization of the file over the wire. The difference above isn't enough to call gRPC faster (it's faster than JSON but it's not faster than pure rust)... So last test.. what if I stream it through a live http upload in Rust? Then both have the same latency from what would in theory be over the wire. Results: Nearly the same... although edged by FastWARC here... Anyway, let me organize this and show you numbers you can easily reproduce. But if you add this, I'd run it on AWS standalone. |
|
What kind of machine is that and what OS? When I run it here on an Ubuntu machine with a Threadripper 2920X (not the newest, but still a powerful CPU), I get only 300MiB/s throughput both when reading from disk and when reading from a tmpfs. Usually, the latter is significantly faster, unless the file is already fully in the page cache (and often even then). Here I'm getting exactly the same throughput in both scenarios and both are an order of magnitude from what you're measuring or what I get from FastWARC alone.
Well, you're certainly limited by your duplex connection, but there's probably also some sender/consumer contention, at least when the server runs in-process. Also keep in mind that if the client isn't also the supplier of the data, you're opening an entirely new security domain. Now you suddenly have to deal with access permissions to prevent streaming of arbitrary protected files from the server system. |
- Raise HTTP/2 flow-control windows to 32 MiB connection / 16 MiB stream, 1 MiB max frame, TCP_NODELAY, 16 MiB message caps, applied on both peers via the new transport module. Adaptive BDP windows stay off: they override fixed windows and intermittently stalled streams saturated in both directions. - Decode bulk bytes fields as bytes::Bytes and parse request chunks in place through a BufRead ChannelReader, removing the per-chunk Vec and BufReader copies on the receive path. - Pack response events into batch messages (64 events, 2 MiB flush) instead of one message per event. - Reject archive_path with PERMISSION_DENIED unless enabled at startup (FASTWARC_GRPC_ALLOW_LOCAL_FILES=1, or WarcParser::with_local_files when embedding), with a test. - Benchmark: parse-only default matching the fastwarc bench, 64 KiB upload chunks, FASTWARC_GRPC_JOBS/LOCAL/URL legs, a rawuds raw-socket floor binary, and results plus methodology in the README.
|
tldr- The best takeaway for this is the ability to scale the machine with gzip (makes sense because it minimized the throughput over the wire) - 16 threads on my machine allowed for 16GB/s of processing. AI slop warning: This link provides the report in more detail. (this is a human typing, but that link is all AI including a recommended slop it wanted me to paste to you) The report does do an exceptional job to show we're now squeezing all we can out of the machine. |
|
I'm now getting 927.5 MiB/s with tmpfs and 753.4 MiB/s with SSD read. That's better, but still not what you're reporting. FASTWARC_GRPC_JOBS=16 hangs for a while, then reports 3386.0 MiB/s. Haven't looked into that enough yet to understand what exactly it does and why we cannot get the normal full print output. |
FASTWARC_GRPC_JOBS>1 previously printed nothing until the final summary, which reads as a hang on longer runs. Progress lines now come from shared counters every 500 ms and cover all streams combined; the single-stream output format is unchanged.
b1cea37 to
dc84590
Compare
|
I'm going to keep trying to optimize the speed further. The speed difference makes sense - my SSDs are a little past the insanity point for speed and the per-core speed is slightly faster (I'd rather have a threadripper though!). I can do a live recording of the timings, it's probably the IO that makes it faster. I want it to match the speed as much as I can. Can you try FASTWARC_GRPC_JOBS=12 on the gzip file rather than 16 on uncompressed? Uncompressed jobs hit the memory-bandwidth plateau by design. So this would actually show a good performance use case for grpc. |
|
With that, it's reporting 7680.6 MiB/s. But something's wrong. The entire file is about 5GiB uncompressed. So by that statistic, it should have taken around 0.7s to parse it (faster than single-core plain FastWARC), whereas in reality, it took 8.3s. |
With FASTWARC_GRPC_JOBS=N each stream parses the full archive, so the totals cover N times the file. Say so up front and add a per-stream average line to the summary, so aggregate figures are not mistaken for single-file wall time.
|
That's exactly what I expected.. that's 8.3s for 12 jobs.. Yes, it takes longer but the compression has enough CPU free for the gzip IO churning that you end up with being able to recover the overall throughput by running 12 jobs concurrently. What it means is that concurrency win isn't gRPC vs. local - it's that the wire cost nothing for overall throughput if you stack the streams. Which in a pipeline is exactly how I'd would approach it. |
|
I'm going to test another approach - the gzip format will allow us to multiplex the file into multiple threads for processing and stream the results back concurrently. I think this can solve your gzip issue in general - the file format makes it possible to do. I'll test that now. |
Yes, but it also means that the per-core speed is still only 640MiB/s. That is fine for Gzip, but it is pretty unacceptable for uncompressed WARCs (even though that's more of an artificial benchmarking scenario). With 12 parallel FastWARC processes, I could probably achieve 60GiB/s, so the numbers aren't comparable. I appreciate that the gRPC server makes it easy to parallelise the streaming. That is useful, but for raw throughput, we should compare single-core performance, otherwise we're comparing apples and oranges. That said, thanks so far for putting in the work (or letting Claude put it in, whatever). We can probably optimise this more later. My experience from writing FastWARC is that AI is certainly useful to make focused changes and optimisations on your behalf when you know exactly what to expect, but when it comes to squeezing out the last few seconds, it consistently failed. No matter what micro-optimisations it proposed, the resulting code always ran slower than before, never faster. So, optimising FastWARC this far was definitely a lot of manual work and I wouldn't expect this to be different here. I will start reviewing the rest of the code next week and add comments. Feel free to experiment further in the meantime.
Link is dead. |
|
Regarding the claude write up- I reread it and although it read well, but it was grandstanding too much so I took it down. Sorry about that. I know I can make it faster but I need to first measure more. I have a clear understanding of the socket overhead - which seems to be around 50%. After that, the actual grpc ops are fast. I think it's memory copy churn with creating the grpc responses that are causing it. I think network is about as good as it can be. With other grpc server solutions, I've often added an HTTP PUT input on the grpc server and always have a streaming output. That would be faster and more effective than a grpc client without. the memory overhead I'll keep trying.. |
|
OK - I think I found a way to make it faster. But I ran this through the entire stack on the machine - and we're up against system IO calls and input starvation. There is a ton of memory churn, but it's not what's causing the speed problems. It's still worth optimizing 2 layers of it. The tonic codec copies every chunk twice: once into the HTTP/2 frame buffer on send, and once merging frames back into a buffer on receive. Neither copy is required by the wire format, so I added a wire-compatible route that hands the bytes to HTTP/2 by reference and decodes them by reference on the other side. That recovers 10-20% depending on concurrency. Also if you run the on-disk test multiple times, most of that file lives in the page cache. You're not going to see any of the latency that comes with sockets. Then the biggest 2 taxes. First, if you put anything over a unix socket, you pay 2 memcopy operations, one into the kernel and one out. There's no way around this, and it's where most of the slowdown lives: feeding fastwarc through a bare socket with no protocol at all already costs ~43% on my machine. Second, input starvation. The socket-fed parser runs at about 90% of what the socket itself can move, so the parser is sitting idle waiting for bytes, and I don't think there's a way around that either. I'll demonstrate everything above with tests. The good and bad news is that this is about as fast as it gets, because these are system-level limits that on-disk parsing never has to touch. But that's also why parallelizing works: the per-stream socket cost stops mattering once you stack streams, so we can recover most of the overall throughput. Does that make sense? The tests I'm writing should demonstrate this. |
|
OK I pushed the latest testing suite.. this one should tell you the speed for each layer. I have attached my output. |
|
Going to look at this again today too.. I have a little time. |
phoerious
left a comment
phoerious
left a comment
There was a problem hiding this comment.
First rough round of reviews.
Every gRPC service in the workspace defaulted to 50051, which collides when several run on one host. Assign fastwarc-grpc the unique default port 50061 (FASTWARC_GRPC_ADDR fallback, client/example defaults, and docs) to avoid collision with sibling services.
Centralize server defaults and hard limits, align the explicit HTTP guard with ArchiveIterator, and document payload-length and batching behavior. Add focused coverage for local-path errors, batch limits, concurrent streams, disabled detection, and omitted payloads.
|
Thanks for the changes. I'm away for two weeks. I'll continue reviewing afterwards. |
|
2 participants
Follow-up to #57 adds
fastwarc-grpcas a separate workspace crate, with documentation, tests, benchmarks, and CI coverage.What is included
fastwarc-grpccrate with a path dependency on the siblingfastwarc-rscrate. It inherits workspace package metadata and remains outside the default members because stub generation requiresprotoc.ParseWarcand unaryParseArchiveRPCs with shared parsing, filtering, digest verification, payload decoding, and error handling.FASTWARC_GRPC_ALLOW_LOCAL_FILES=1because they allow clients to read files from the server.benchmarks/warc/fastwarc-grpc. Its README records the reviewer reproduction and distinguishes single-stream results from aggregate multi-stream throughput.fastwarc-grpc. Each runner installs the matching precompiledprotocarchive.Local validation
The current PR head passes:
cargo test -p fastwarc-grpc: 48 tests, including 7 library tests, 38 integration tests, and 3 doctestscargo fmt -p fastwarc-grpc --checkcargo clippy -p fastwarc-grpc --all-targets --no-deps -- -D warnings -D clippy::pedanticRUSTDOCFLAGS="-D warnings" cargo doc -p fastwarc-grpc --no-depsbuf lintandbuf format --diff --exit-codefor the protobuf definitionsThe tests reuse existing repository fixtures, including
fastwarc-rs/tests/fixtures/warcfile.warc.zst.