Skip to content

Commit 2de356a

Browse files
xiaoxichenxiaoxichen
authored
SDSTOR-21407 Make per-service dev_type and size_pct configurable in i… (#411)
* SDSTOR-21407 Make per-service dev_type and size_pct configurable in init_homestore Add HSDevTypeOverride enum and per-service {meta,log,index,replication}_{dev_type,size_pct} fields to HSBackendSettings. Defaults are AUTO/-1.0 so existing hybrid/single mode logic is fully preserved. When a config value is set it overrides the built-in default via resolve_dev_type() and resolve_size_pct() helpers. Also tune standalone (single-device) defaults: meta/index 5%- >1%, replication 79%->87%. Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com> * Address comment Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
1 parent 88b936f commit 2de356a

4 files changed

Lines changed: 416 additions & 31 deletions

File tree

‎conanfile.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010

1111
class HomeObjectConan(ConanFile):
1212
name = "homeobject"
13-
version = "4.1.10"
13+
version = "4.1.11"
1414

1515
homepage = "https://github.com/eBay/HomeObject"
1616
description = "Blob Store built on HomeStore"

‎docs/adr/blob-index-analyze.md‎

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
# HomeObject Blob Index Analyze
2+
3+
**Date:** 2026-05-06
4+
**Purpose:** Reference for blob index sizing and capacity planning
5+
6+
---
7+
## Execution Summary
8+
9+
### Scenario 1: 20TB Data with 2GB Memory Budget
10+
11+
**Configuration:**
12+
- Total data: 20TB
13+
- `io_mem_size`: 2GB
14+
- **Fixed cache pool:** 2GB × 65% = **1.3 GB** (always allocated)
15+
16+
| Blob Size | Blob Count | Index Disk | Hard Memory | Cache Pool | Cache Coverage |
17+
|-----------|------------|------------|-------------|------------|----------------|
18+
| **8 KB** | 2.56B | 60.1 GB | 117 MB | 1.3 GB | **2.16%** ⚠️ |
19+
| **256 KB**| 80M | 1.88 GB | 3.67 MB | 1.3 GB | **69%** ✓ |
20+
21+
22+
---
23+
24+
### Scenario 2: 128TB Data with 2GB Memory Budget
25+
26+
**Configuration:**
27+
- Total data: 128TB
28+
- `io_mem_size`: 2GB
29+
- **Fixed cache pool:** 2GB × 65% = **1.3 GB** (always allocated)
30+
31+
| Blob Size | Blob Count | Index Disk | Hard Memory | Cache Pool | Cache Coverage |
32+
|-----------|------------|------------|-------------|------------|----------------|
33+
| **8 KB** | 16.4B | 385 GB | 770 MB | 1.3 GB | **0.34%** ⚠️⚠️ |
34+
| **256 KB**| 512M | 12.0 GB | 23.4 MB | 1.3 GB | **10.8%** ✓ |
35+
36+
37+
---
38+
39+
## Action Items
40+
### Shrink down the size of Index_vdev
41+
#### [HDD SKU]
42+
Now we use 45% of META drive (200GB*45% =90GB) on HDD SKUs. The number is generally correct as worst case we can consume up to 60GB.
43+
Shrinking it down from 90GB to 60GB saves 60MB memory which is negligible.
44+
#### [QLC SKU]
45+
QLC node total data capacity: ~128 TB.
46+
5% of QLC size (114 TB × 5% = 5721.96 GB) consumes ~11 GB memory for allocator.
47+
Changing it to 0.5% based on below calculation, as a result, Index size would be ~500GB with Hard Memory ~1GB.
48+
Also, we probably should tune down the log pct from 10% to 0.5% (570GB) as well, but this tuning should be done via config
49+
to avoid failing the (vdev too small)
50+
51+
```
52+
Worst case assuming blob size is 8KB, the Index size should be ~0.3% of DataSize
53+
54+
(DataSize/8K)*(4KB/167)*1.01 ==> DataSize * (4K/8K)/167 * 1.01 = DataSize * 0.5/167 * 1.01 ≈ DataSize * 0.3%.
55+
```
56+
### Configure mem_size for QLC nodes.
57+
58+
For QLC nodes we have ~35GB+ memory for a SM, bumping the app_mem_size allows more index be cached in memory, however,
59+
the performance gap is still subject to performance evaluation.
60+
61+
## Quick Estimation Formulas
62+
63+
**For any blob size and data volume:**
64+
65+
```
66+
1. Blob count (N) = Data size / Avg blob size
67+
68+
2. Index disk space ≈ (N / 167) × 4KB × 1.01
69+
(1.01 factor accounts for internal nodes)
70+
71+
3. Hard memory = Index disk / 4096 × 8 bytes
72+
(Allocator) (Always resident, non-negotiable)
73+
74+
4. Soft memory = min(io_mem_size × 65%, Index disk × 10%)
75+
(Working set) (Capped by global memory budget; "Index disk × 10%" in tables
76+
is a conservative working-set estimate, not the actual cap)
77+
78+
5. Dirty buffer limit = io_mem_size × 10%
79+
(Transient peak) (Shared across all writes, not index-specific)
80+
```
81+
82+
**Memory Components:**
83+
- **Hard (Allocator):** Always resident in `folly::MPMCQueue`, holds ALL free blocks
84+
- Independent of `io_mem_size`, scales with index vdev size
85+
- Formula: `(index_disk / 4096) × 8 bytes` (4B `blk_num_t` + 4B `atomic<uint32_t>` sequence per slot)
86+
87+
- **Soft (Working Set):** LRU-managed btree node cache
88+
- **Capped by:** `io_mem_size × cache_size_percent / 100` (default 65%)
89+
- **Shared by:** All indexes + data service caches
90+
- **Not capped by:** Index vdev size
91+
92+
- **Transient (Dirty):** Temporary write buffers
93+
- **Capped by:** `io_mem_size × dirty_buf_percent / 100` (default 10%)
94+
- Freed after checkpoint flush
95+
96+
---
97+
## Quick Reference
98+
99+
| Node Type | Fanout | Entry Size | Calculation |
100+
|-----------|--------|------------|-------------|
101+
| **Leaf** | **167** | 24 bytes | (4096 - 72) / 24 = 167.67 → 167 |
102+
| **Internal** | **125** | 32 bytes | (4096 - 72) / 32 = 125.75 → 125 |
103+
104+
---
105+
106+
## Detailed Breakdown
107+
108+
### Node Structure (4KB nodes)
109+
110+
```
111+
Total node size: 4096 bytes
112+
Header size: 72 bytes (persistent_hdr_t)
113+
Data area: 4024 bytes
114+
```
115+
116+
### Leaf Nodes (Store blob → physical address mappings)
117+
118+
**Entry Structure:**
119+
```cpp
120+
Key: BlobRoute (shard_id + blob_id) = 16 bytes
121+
Value: MultiBlkId (physical block addr) = 8 bytes
122+
Total: = 24 bytes
123+
```
124+
125+
**Capacity:**
126+
```
127+
Fanout = floor(4024 / 24) = 167 entries per leaf
128+
Wasted space = 4024 - (167 × 24) = 16 bytes (99.6% utilization)
129+
```
130+
131+
### Internal Nodes (Store routing keys → child pointers)
132+
133+
**Entry Structure:**
134+
```cpp
135+
Key: BlobRoute (separator key) = 16 bytes
136+
Value: BtreeLinkInfo (child pointer) = 16 bytes
137+
- bnodeid_t (child node ID) = 8 bytes
138+
- link_version (concurrency ctrl) = 8 bytes
139+
Total: = 32 bytes
140+
```
141+
142+
**Capacity:**
143+
```
144+
Fanout = floor(4024 / 32) = 125 children per internal node
145+
Wasted space = 4024 - (125 × 32) = 24 bytes (99.4% utilization)
146+
```
147+
148+
---
149+
150+
## Key Details
151+
152+
### Type Definitions
153+
```cpp
154+
// From HomeObject
155+
using shard_id_t = uint64_t; // 8 bytes
156+
using blob_id_t = uint64_t; // 8 bytes
157+
using bnodeid_t = uint64_t; // 8 bytes (HomeStore)
158+
159+
#pragma pack(1)
160+
struct BlobRoute {
161+
shard_id_t shard; // 8 bytes
162+
blob_id_t blob; // 8 bytes
163+
}; // Total: 16 bytes (packed)
164+
```
165+
166+
### Internal Node Keys
167+
- Internal nodes store **actual BlobRoute keys** (same as leaf keys)
168+
- Keys are **separator keys**: the **last key** from the left child subtree
169+
- NOT synthetic/aggregate keys - real blob identifiers
170+
171+
### Index Sizing Formula
172+
173+
For N blobs:
174+
```
175+
Leaf nodes = ceil(N / 167)
176+
Level 1 = ceil(Leaf nodes / 125)
177+
Level 2 = ceil(Level 1 / 125)
178+
...
179+
Total size = Total nodes × 4096 bytes
180+
```
181+
182+
**Example:** 1 million blobs
183+
```
184+
Leaf: 5,989 nodes
185+
Level 1: 48 nodes
186+
Level 2: 1 node (root)
187+
Total: 6,038 nodes = 23.59 MB
188+
```
189+
190+
---
191+
192+
## Node Implementation
193+
194+
- **Node Type:** `SimpleNode` (fixed-size key/value pairs)
195+
- **Packing:** Sequential, no per-entry overhead
196+
- **Layout:** `[K₀V₀][K₁V₁][K₂V₂]...` (tightly packed)
197+
- **Alignment:** None - entries packed contiguously
198+
199+
---
200+
201+
## Memory Overhead
202+
203+
For index vdev size S:
204+
- **Allocator overhead:** ~0.2% of S
205+
- Uses `FixedBlkAllocator` (4KB fixed block size)
206+
- Maintains all free blocks in memory via `folly::MPMCQueue`
207+
- Each slot: 8 bytes (4B `blk_num_t` + 4B `atomic<uint32_t>` sequence)
208+
- For 10GB index: ~20 MB allocator memory
209+
210+
**Total overhead per blob:** ~24-25 bytes (including tree structure overhead)
211+
212+
---
213+
214+
## Production Scaling Analysis
215+
216+
### Memory Cap Model
217+
218+
**Global Memory Budget:** `io_mem_size` (configuration parameter)
219+
```cpp
220+
// From homestore_decl.hpp
221+
uint64_t io_mem_size() const {
222+
return (hugepage_size != 0) ? hugepage_size : app_mem_size;
223+
}
224+
```
225+
226+
**Index Working Set Cap:**
227+
```cpp
228+
// From resource_mgr.cpp:179
229+
max_index_cache = io_mem_size × cache_size_percent / 100
230+
= io_mem_size × 65% (default from homestore_config.fbs)
231+
```
232+
233+
**Dirty Buffer Cap:**
234+
```cpp
235+
// From resource_mgr.cpp:233
236+
max_dirty_buffers = io_mem_size × dirty_buf_percent / 100
237+
= io_mem_size × 10% (default)
238+
```
239+
240+
**Key Points:**
241+
1. **Global cap trumps index size:** Working set is limited by `io_mem_size × 65%`, NOT index vdev size
242+
- Example: 10GB index but `io_mem_size = 1GB` → max cache = 650MB (not 6.5GB)
243+
- LRU evictor (created in homestore.cpp:280) enforces this limit
244+
245+
2. **Allocator memory is separate:** Not counted in cache budget, always required
246+
- Hard memory = ~0.2% of index disk (8 bytes per 4KB block)
247+
248+
3. **Shared budget:** `io_mem_size` serves ALL HomeStore components (index, data, log)
249+
- Index cache competes with data cache for this budget
250+
- Multi-index deployments share the same `cache_size_percent` pool
251+
252+
**In the tables below:**
253+
- "Soft Memory" shows 65% of index disk (theoretical maximum)
254+
- Actual working set = `min(index_disk × 65%, io_mem_size × 65%)`
255+
- Configure `io_mem_size` large enough to avoid excessive cache thrashing
256+
257+
---
258+
259+
### Case 1: 256KB Blobs
260+
261+
| Data Size | Blob Count | Index Disk | Hard Memory (Allocator) | Soft Memory (Working Set)* |
262+
|-----------|------------|------------|-------------------------|----------------------------|
263+
| **1 TB** | 4.0M | 95.9 MB | 187 KB | 10 MB (10% of index) |
264+
| **20 TB** | 80.0M | 1.88 GB | 3.67 MB | 188 MB (10% of index) |
265+
| **128 TB** | 512.0M | 12.0 GB | 23.4 MB | 1.2 GB (10% of index) |
266+
267+
\* *Actual = min(shown value, `io_mem_size × 65%`). Configure `io_mem_size` accordingly.*
268+
269+
**Calculation:**
270+
```
271+
Blob count = Data size / 256KB
272+
Index disk = ceil(Blob count / 167) × 4KB + overhead (3-level tree)
273+
Hard memory = Index disk / 4KB blocks × 12 bytes (allocator queue)
274+
Soft memory = min(Index disk × 10%, io_mem_size × 65%)
275+
```
276+
277+
---
278+
279+
### Case 2: 8KB Blobs (Worst Case)
280+
281+
| Data Size | Blob Count | Index Disk | Hard Memory (Allocator) | Soft Memory (Working Set)* |
282+
|-----------|------------|------------|-------------------------|----------------------------|
283+
| **1 TB** | 128.0M | 3.01 GB | 5.88 MB | 0.3 GB (10% of index) |
284+
| **20 TB** | 2.56B | 60.1 GB | 117 MB | 6 GB (10% of index) |
285+
| **128 TB** | 16.4B | 385 GB | 770 MB | 38 GB (10% of index) |
286+
287+
\* *Actual = min(shown value, `io_mem_size × 65%`). Large deployments need substantial `io_mem_size`.*
288+
289+
**Calculation:**
290+
```
291+
Blob count = Data size / 8KB
292+
Index disk = ceil(Blob count / 167) × 4KB + overhead (4-5 level tree)
293+
Hard memory = Index disk / 4KB blocks × 12 bytes (allocator queue)
294+
Soft memory = min(Index disk × 10%, io_mem_size × 65%)
295+
```
296+
297+
---
298+
299+
300+
301+
## References
302+
303+
- **HomeStore:** `/Users/xiaoxchen/Code/HomeStore`
304+
- Btree: `src/include/homestore/btree/detail/simple_node.hpp`
305+
- Block allocator: `src/lib/blkalloc/fixed_blk_allocator.{h,cpp}`
306+
307+
- **HomeObject:** `/Users/xiaoxchen/Code/HomeObject`
308+
- Index KV: `src/lib/homestore_backend/index_kv.hpp`
309+
- Blob route: `src/lib/blob_route.hpp`
310+
311+
---
312+
313+
**Last Updated:** 2026-05-06
314+
**Validated Against:** HomeStore commit d9cf5c10, HomeObject current
315+
**Includes:** Production scaling for 1TB-128TB with 256KB and 8KB blob sizes

‎src/lib/homestore_backend/hs_backend_config.fbs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ namespace homeobjectcfg;
55
attribute "hotswap";
66
attribute "deprecated";
77

8+
// Per-service device type override for format_and_start.
9+
// AUTO (default) defers to the built-in hybrid/single mode logic.
10+
enum HSDevTypeOverride : uint8 { AUTO = 0, FAST = 1, DATA = 2 }
811

912
table HSBackendSettings {
1013
// timer thread freq in us
@@ -50,6 +53,21 @@ table HSBackendSettings {
5053
// Check traffic ready before get, default true, set false for 1-replica workaround mode to allow read
5154
check_traffic_ready_before_get: bool = true;
5255

56+
// Per-service device type and size_pct overrides for format_and_start.
57+
// dev_type defaults to AUTO (0 = defer to hybrid/single code logic).
58+
// size_pct defaults to -1.0 (not-set); any value >= 0 overrides the built-in default.
59+
meta_dev_type: HSDevTypeOverride = AUTO;
60+
meta_size_pct: float = -1.0;
61+
62+
log_dev_type: HSDevTypeOverride = AUTO;
63+
log_size_pct: float = -1.0;
64+
65+
index_dev_type: HSDevTypeOverride = AUTO;
66+
index_size_pct: float = -1.0;
67+
68+
replication_dev_type: HSDevTypeOverride = AUTO;
69+
replication_size_pct: float = -1.0;
70+
5371
}
5472

5573
root_type HSBackendSettings;

0 commit comments

Comments
 (0)