Skip to content

Commit d7d2ebb

Browse files
cbb330cbb330
authored
Add explicit build/test requirements for tasks 32-35 (apache#90)
Updated project_spec.txt and task_list.json to require: - CMake configure with -DARROW_ORC=ON -DARROW_DATASET=ON - Build arrow_dataset target - Run C++ ORC dataset tests (ctest -R orc) - Run Python E2E tests (test_orc.py and test_dataset.py -k orc) Tasks 32-35 now include build_and_test_requirements section with mandatory steps that MUST pass before marking task complete.
1 parent 713a8a5 commit d7d2ebb

2 files changed

Lines changed: 138 additions & 3 deletions

File tree

‎project_spec.txt‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
<project_context>
2+
<project_name>Apache Arrow - ORC Predicate Pushdown</project_name>
3+
4+
<overview>
5+
LinkedIn fork of Apache Arrow implementing ORC predicate pushdown for the Dataset API.
6+
This feature filters ORC file stripes based on column statistics, reducing I/O for selective queries.
7+
</overview>
8+
9+
<technology_stack>
10+
- Language: C++17, Python
11+
- Build: CMake 3.16+, Ninja (preferred), Make
12+
- Compilers: clang++ or g++
13+
- Test framework: Google Test (C++), pytest (Python)
14+
- Arrow tools: archery (test runner)
15+
</technology_stack>
16+
17+
<project_structure>
18+
- cpp/src/arrow/dataset/file_orc.h - ORC file format header
19+
- cpp/src/arrow/dataset/file_orc.cc - ORC file format implementation
20+
- cpp/src/arrow/dataset/file_orc_test.cc - ORC dataset tests
21+
- cpp/src/arrow/adapters/orc/ - Low-level ORC adapter
22+
- python/pyarrow/tests/test_orc.py - Python ORC tests
23+
</project_structure>
24+
25+
<how_to_build>
26+
# Configure (one-time setup if build directory doesn't exist)
27+
cd cpp
28+
mkdir -p build && cd build
29+
cmake .. \
30+
-DCMAKE_BUILD_TYPE=Debug \
31+
-DARROW_ORC=ON \
32+
-DARROW_DATASET=ON \
33+
-DARROW_BUILD_TESTS=ON \
34+
-DARROW_PARQUET=ON
35+
36+
# Build the ORC dataset target (from cpp/build directory)
37+
cmake --build . --target arrow_dataset -j4
38+
39+
# Or build specific test executable
40+
cmake --build . --target arrow-dataset-file-orc-test -j4
41+
</how_to_build>
42+
43+
<how_to_test>
44+
# C++ Unit Tests (from cpp/build directory)
45+
ctest --output-on-failure -R orc
46+
47+
# Run specific ORC dataset tests
48+
./debug/arrow-dataset-file-orc-test
49+
50+
# Python Tests
51+
cd python
52+
pytest pyarrow/tests/test_orc.py -v
53+
54+
# E2E / Integration Tests
55+
pytest pyarrow/tests/test_dataset.py -v -k "orc"
56+
</how_to_test>
57+
</project_context>
58+
59+
<current_goal>
60+
Complete ORC predicate pushdown implementation with comprehensive testing.
61+
Code MUST compile and tests MUST pass before marking tasks complete.
62+
</current_goal>
63+
64+
<build_verification>
65+
MANDATORY: BEFORE committing any code changes:
66+
67+
1. Configure CMake with REQUIRED flags (if not already configured):
68+
cd cpp/build
69+
cmake .. -DARROW_ORC=ON -DARROW_DATASET=ON -DARROW_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Debug
70+
71+
2. Build the target:
72+
cmake --build . --target arrow_dataset -j4
73+
74+
3. Run C++ ORC dataset tests:
75+
ctest --output-on-failure -R orc
76+
# OR run directly:
77+
./debug/arrow-dataset-file-orc-test
78+
79+
4. Run Python E2E tests:
80+
cd ../python
81+
pytest pyarrow/tests/test_orc.py -v
82+
pytest pyarrow/tests/test_dataset.py -v -k "orc"
83+
84+
5. Fix any compilation errors or test failures BEFORE committing.
85+
86+
CRITICAL: You MUST run ALL of the above (C++ build, C++ tests, Python E2E).
87+
DO NOT commit code that does not compile or fails tests.
88+
DO NOT mark a task complete without passing ALL test suites.
89+
</build_verification>
90+
91+
<constraints>
92+
- ALL code MUST compile without errors
93+
- ALL tests MUST pass before marking a task complete
94+
- Match existing Arrow code style exactly
95+
- Follow Parquet implementation patterns where applicable
96+
- Thread safety is critical - use util::Mutex, not std::mutex
97+
</constraints>

‎task_list.json‎

Lines changed: 41 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -742,7 +742,9 @@
742742
"task": "Add float32/float64 type support",
743743
"description": "Extend statistics support to floating-point types. Handle NaN, infinity, signed zero edge cases.",
744744
"files_to_modify": [
745-
"cpp/src/arrow/dataset/file_orc.cc"
745+
"cpp/src/arrow/adapters/orc/adapter.cc",
746+
"cpp/src/arrow/dataset/file_orc.cc",
747+
"cpp/src/arrow/dataset/file_orc_test.cc"
746748
],
747749
"parquet_reference": {
748750
"notes": "See allium spec FLOATING-POINT EDGE CASES section"
@@ -751,6 +753,14 @@
751753
"Float predicates work correctly",
752754
"NaN handling verified"
753755
],
756+
"build_and_test_requirements": {
757+
"MANDATORY": "You MUST complete ALL of these steps before marking this task complete",
758+
"step_1_configure": "cd cpp/build && cmake .. -DARROW_ORC=ON -DARROW_DATASET=ON -DARROW_BUILD_TESTS=ON",
759+
"step_2_build": "cmake --build . --target arrow_dataset -j4",
760+
"step_3_cpp_tests": "ctest --output-on-failure -R orc",
761+
"step_4_python_e2e": "cd python && pytest pyarrow/tests/test_orc.py -v && pytest pyarrow/tests/test_dataset.py -v -k orc",
762+
"failure_policy": "If ANY test fails, fix before committing. Do NOT skip tests."
763+
},
754764
"status": "pending",
755765
"depends_on": [14],
756766
"priority": "P2"
@@ -761,7 +771,9 @@
761771
"task": "Add string/binary type support",
762772
"description": "Extend to string/binary types. Handle truncation in statistics.",
763773
"files_to_modify": [
764-
"cpp/src/arrow/dataset/file_orc.cc"
774+
"cpp/src/arrow/adapters/orc/adapter.cc",
775+
"cpp/src/arrow/dataset/file_orc.cc",
776+
"cpp/src/arrow/dataset/file_orc_test.cc"
765777
],
766778
"parquet_reference": {
767779
"concept": "String column pushdown test",
@@ -773,6 +785,14 @@
773785
"String predicates work correctly",
774786
"Truncation handled conservatively"
775787
],
788+
"build_and_test_requirements": {
789+
"MANDATORY": "You MUST complete ALL of these steps before marking this task complete",
790+
"step_1_configure": "cd cpp/build && cmake .. -DARROW_ORC=ON -DARROW_DATASET=ON -DARROW_BUILD_TESTS=ON",
791+
"step_2_build": "cmake --build . --target arrow_dataset -j4",
792+
"step_3_cpp_tests": "ctest --output-on-failure -R orc",
793+
"step_4_python_e2e": "cd python && pytest pyarrow/tests/test_orc.py -v && pytest pyarrow/tests/test_dataset.py -v -k orc",
794+
"failure_policy": "If ANY test fails, fix before committing. Do NOT skip tests."
795+
},
776796
"status": "pending",
777797
"depends_on": [14],
778798
"priority": "P2"
@@ -783,7 +803,9 @@
783803
"task": "Add timestamp/date type support",
784804
"description": "Extend to temporal types. Handle unit conversion and timezone issues.",
785805
"files_to_modify": [
786-
"cpp/src/arrow/dataset/file_orc.cc"
806+
"cpp/src/arrow/adapters/orc/adapter.cc",
807+
"cpp/src/arrow/dataset/file_orc.cc",
808+
"cpp/src/arrow/dataset/file_orc_test.cc"
787809
],
788810
"parquet_reference": {
789811
"concept": "Duration column pushdown test",
@@ -795,6 +817,14 @@
795817
"Temporal predicates work correctly",
796818
"Unit conversion correct"
797819
],
820+
"build_and_test_requirements": {
821+
"MANDATORY": "You MUST complete ALL of these steps before marking this task complete",
822+
"step_1_configure": "cd cpp/build && cmake .. -DARROW_ORC=ON -DARROW_DATASET=ON -DARROW_BUILD_TESTS=ON",
823+
"step_2_build": "cmake --build . --target arrow_dataset -j4",
824+
"step_3_cpp_tests": "ctest --output-on-failure -R orc",
825+
"step_4_python_e2e": "cd python && pytest pyarrow/tests/test_orc.py -v && pytest pyarrow/tests/test_dataset.py -v -k orc",
826+
"failure_policy": "If ANY test fails, fix before committing. Do NOT skip tests."
827+
},
798828
"status": "pending",
799829
"depends_on": [14],
800830
"priority": "P2"
@@ -813,6 +843,14 @@
813843
"verification": [
814844
"Nested field predicates work"
815845
],
846+
"build_and_test_requirements": {
847+
"MANDATORY": "You MUST complete ALL of these steps before marking this task complete",
848+
"step_1_configure": "cd cpp/build && cmake .. -DARROW_ORC=ON -DARROW_DATASET=ON -DARROW_BUILD_TESTS=ON",
849+
"step_2_build": "cmake --build . --target arrow_dataset -j4",
850+
"step_3_cpp_tests": "ctest --output-on-failure -R orc",
851+
"step_4_python_e2e": "cd python && pytest pyarrow/tests/test_orc.py -v && pytest pyarrow/tests/test_dataset.py -v -k orc",
852+
"failure_policy": "If ANY test fails, fix before committing. Do NOT skip tests."
853+
},
816854
"status": "pending",
817855
"depends_on": [14],
818856
"priority": "P2"

0 commit comments

Comments
 (0)