Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
7c7094e
PAX: Add initial project structure and core components
jiaqizho Mar 31, 2023
740fadb
Feature: Implement storage layer abstraction with writer/reader inter…
jiaqizho Apr 3, 2023
2101023
PAX: Integrate Google Test (gtest) and Google Mock (gmock)
jiaqizho Apr 3, 2023
0d1245c
PAX: Implement local file system operations
jiaqizho Apr 4, 2023
9568689
Feature: pax storage support tuple insert
gongxun0928 Apr 12, 2023
c31fbcc
Feature: support table scan in pax storage
gongxun0928 Apr 14, 2023
737c6d1
Feature: linking libpostgres.so and add some unit testing
gongxun0928 Apr 19, 2023
cebc0c7
bugfix: block_id is empty in catalog table when insert tuple
gongxun0928 Apr 20, 2023
a0d1f0c
PAX: support truncate operation
Apr 16, 2023
9f78479
Feature: catalog add the ptblocksize attribute
gongxun0928 Apr 21, 2023
6e6f3d1
Read include path from pg_config instead of hardcode.
my-ship-it Apr 21, 2023
dc3dc0d
Update dml_init and dml_fini hooks
gfphoenix78 Apr 24, 2023
0c7f990
Split table access methods into 2 classes, as static class functions
gfphoenix78 Apr 24, 2023
67b4502
Feature: Implement RelationSize and EstimateRelSize
Apr 25, 2023
ea1f697
Feature: support table rescan in pax storage
Apr 26, 2023
a61893e
PAX: format headers and their order
Apr 27, 2023
d1ebbe8
Feature: implemention anlayze and samping AM interface
May 6, 2023
92371e2
PAX: Add reloptions support
gfphoenix78 May 8, 2023
b4e1b07
PAX: split namespace to pax and paxc
gfphoenix78 May 8, 2023
1dd8c1f
Feature: Pax support bulk insert operation
May 8, 2023
0aa0bd9
Feature: port ORC into PAX
jiaqizho May 10, 2023
c136638
Fix cmake 3.11 will got error cause POSITION_INDEPENDENT_CODE relation
jiaqizho May 11, 2023
8be8e9d
Refactor scanner: merge scanner and PaxScanDesc into PaxScanDesc
gfphoenix78 May 10, 2023
7d30e8c
PAX: Add exception catch for some C++ callback
gfphoenix78 May 11, 2023
f49fd87
Generate SQL to use fixed OID
gfphoenix78 May 12, 2023
e3c8e47
Feature: PAX support set new tablespace operation
May 12, 2023
267ebab
Bugfix: remove useless gp_debug_linger timewait for debug usage in GTEST
May 12, 2023
c7cdd1f
Bugfix: select empty table raise panic
gongxun0928 May 15, 2023
8332b56
PAX: Tuple will in-place updated, so should not keep the memory ptr
jiaqizho May 16, 2023
a7f0cd7
Feature: add PAX CI build & UT pipeline
May 13, 2023
dcd9320
Bugfix: Create mircro-partition table directory for trancate table case
May 17, 2023
2453d56
Fix memory problem caused by DatumFromCString
jiaqizho May 17, 2023
34c3088
PAX: Better orc seek function
jiaqizho May 19, 2023
219cd9e
PAX: Better non fixed column
jiaqizho May 23, 2023
6121b1f
PAX: Run CI job automatically on push
May 23, 2023
108272f
Fix CI without branch
jiaqizho May 24, 2023
71a2f3f
Feature: table writer support split strategy
jiaqizho May 25, 2023
35837f7
Fix `SeekTuple` logic error
jiaqizho May 25, 2023
4c6fe7f
Feature: MicroPartitionReader support reused data buffer in scan
jiaqizho May 25, 2023
fe67eb7
Fix CException won't be catched in cpp
jiaqizho May 30, 2023
906519c
Feature: introduce gmock replace simple mock
jiaqizho May 31, 2023
d9f076c
Feature: support more pg-types in pax storage
gongxun0928 Jun 2, 2023
5acd43c
Fix pax can not deal non-virtual-tuple
jiaqizho May 31, 2023
eeb64d5
Feature: support delete/update in pax storage
gongxun0928 Jun 9, 2023
a8f4f36
Feature: pax support null field
jiaqizho Jun 13, 2023
6e0b2d7
Fix analyze table failed when pax table has null and non-null column …
gongxun0928 Jun 15, 2023
468be67
Support create empty table which do not contains any columns
gongxun0928 Jun 15, 2023
b61273b
Enhancement: Replace the binary path of protobuf
gongxun0928 Jun 16, 2023
87c46ae
Query with SPI will call the ExecutorEnd multiple times in recursive …
gongxun0928 Jun 16, 2023
08d2a86
Optimize: defined basic code style & Introduce to cpp-tidy
jiaqizho Jun 19, 2023
e491be8
Optimize: enable clang-tidy in CI
jiaqizho Jun 19, 2023
49d30bd
Feature: Support `add column` ddl in PAX
Jun 20, 2023
07ca734
clang-tidy needs to skip files removed by git rm
gongxun0928 Jun 21, 2023
1d6b626
Fix: byte alignment is necessary when writing data for non-fixed columns
gongxun0928 Jun 21, 2023
d9bef3d
Feature: support Mirco-paritiition non-transactional delete
Jun 26, 2023
dbc8ff5
Add clang-tidy.result into CI artifacts for debugging usage
Jun 20, 2023
173ef39
Bugfix: Fix drop column update issue
Jun 29, 2023
b1355a6
Fix: insert data into the external table will crashes
gongxun0928 Jun 29, 2023
8407b44
bugfix: Fixed the error in clang-tidy which the modified file list wa…
gongxun0928 Jun 29, 2023
20dd063
Optimize: remove non-continuous write
jiaqizho Jul 4, 2023
5ff7994
Install pax as a second-party extension
gfphoenix78 Jul 5, 2023
c2daedb
bugfix: fix the incorrect usage of temporary parameter, avoid memory …
gongxun0928 Jul 6, 2023
cfbbc6c
Add a callback for swap_relation_files for PAX
gfphoenix78 Jul 7, 2023
3f4dd23
Fix SwapRelationFiles that should hold relation lock
gfphoenix78 Jul 7, 2023
ea29a56
Feature:introduce pax memory context
jiaqizho Jul 12, 2023
e74a376
Feature: Implement pax access method ScanRescan API
Jul 12, 2023
69786a0
Optimize: ignore paxc_* file in CI clang-tidy check
jiaqizho Jul 13, 2023
45ca845
Always use the database's default tablespace for the auxiliary relati…
Jul 18, 2023
6acb59d
Implement an empty PaxAccessMethod::RelationVacuum to able to run VAC…
Jul 20, 2023
bfc2738
Fix CI will failed in dev branch
jiaqizho Jul 21, 2023
0d4d6ac
Feature/pax format adapted to storage am
gongxun0928 Jul 24, 2023
d87838a
Enhancement: IO functions throw exception instead of returning error …
Jul 28, 2023
c5bbb3c
Feature: Introduce encoding && compress interface in pax column
jiaqizho Jul 30, 2023
efefe20
Fix: kExTypeFileOperationError exception missing error message
jiaqizho Aug 1, 2023
91ba84e
Feature: RLE decoding support template
jiaqizho Aug 2, 2023
6fa4f80
Feature: add PAX projection filter functionality
Aug 8, 2023
f568683
CMake: allow release build if -DENBALE_DEBUG=off
jiaqizho Aug 8, 2023
678ccda
Enhancement: pax extension uses the 1X_STABLE_CP_FEATURE_PAX branch o…
gongxun0928 Aug 9, 2023
6c02c31
Bugfix: Fix ReadTuple in case scan analyze without projection info
Aug 9, 2023
c584f46
Optimize: New implements column projection in orc
jiaqizho Aug 9, 2023
d686334
Enhancement: Remove unnecessary interface from filesystem class
gongxun0928 Aug 10, 2023
c08b7d2
Enhancement: reconstruct the directory structure of pax extension
gongxun0928 Aug 10, 2023
13e8cbf
Feature: pax column support encoding/decoding
jiaqizho Aug 10, 2023
4b9b11e
Bugfix: remove unnecessary files when compiling libpaxformat.so
gongxun0928 Aug 10, 2023
9b45231
Feature: orc support encoding column
jiaqizho Aug 11, 2023
f7a61bd
Rewrite callback implementation of analyze & cleanup code
gfphoenix78 Aug 16, 2023
76d2043
Simplify the iterator interface and cleanup some unused code
Aug 21, 2023
81b4f2c
Feature: Add pax catalog statistics support
Aug 22, 2023
6ef8ba0
Feature: Implement API CopyForCluster
Aug 22, 2023
466ca86
CI: change back to cbdb feature-pax branch
jiaqizho Aug 22, 2023
1499e8a
Feature: Add support for micro-partition-level filter
gfphoenix78 Aug 16, 2023
cc8e2cc
Fix: fs test will got some permissions errors
jiaqizho Aug 25, 2023
7385b1c
Feature: interface adjustment to support vec implements
jiaqizho Aug 29, 2023
0bc3556
Feature: PAX support VEC executor
jiaqizho Aug 29, 2023
c52bb69
Fix: some am functions have no catch common cexception
jiaqizho Aug 29, 2023
a497b81
Op: Remove previous check before delete
jiaqizho Aug 29, 2023
c58e7af
Fix: Build got some error with -DBUILD_GTEST=OFF
jiaqizho Sep 1, 2023
96d1a16
Op: remove OrcIteratorReader
jiaqizho Sep 1, 2023
bba60fd
Op: Add a new vectorization MicroPartitionReader to replace ReadVecTuple
jiaqizho Sep 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions contrib/pax_storage/.ci/tf/qingcloud-provider.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
variable "qingcloud_access_key" {
sensitive = true
type = string
}

variable "qingcloud_secret_key" {
sensitive = true
type = string
}

variable "qingcloud_zone" {
default = "pek3c"
}

variable "instance_name" {
default = "ci"
}

variable "instance_image" {
default = "img-qbpas5m2"
}

variable "instance_class" {
default = 202
}

variable "instance_cpu" {
default = 16
}

variable "instance_memory" {
default = 16384
}

variable "instance_os_disk_size" {
default = 100
}

variable "instance_vxnet" {
default = "vxnet-5tjdylj"
}

variable "instance_keypair" {
default = [
"kp-o07unn26"]
}

terraform {
required_providers {
qingcloud = {
source = "HashDataInc/qingcloud"
version = "1.2.7"
}
ansible = {
source = "nbering/ansible"
version = "1.0.4"
}
}
}

provider "qingcloud" {
access_key = var.qingcloud_access_key
secret_key = var.qingcloud_secret_key
zone = var.qingcloud_zone
}


resource "qingcloud_instance" "ci" {
name = var.instance_name
image_id = var.instance_image
instance_class = var.instance_class
cpu = var.instance_cpu
memory = var.instance_memory
os_disk_size = var.instance_os_disk_size
managed_vxnet_id = var.instance_vxnet
keypair_ids = var.instance_keypair
}

resource "ansible_host" "ci" {
inventory_hostname = qingcloud_instance.ci.private_ip
groups = [
"runner"]
vars = {
ansible_user = "root"
}
}
175 changes: 175 additions & 0 deletions contrib/pax_storage/.clang-format
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
---
Language: Cpp
# BasedOnStyle: Google
AccessModifierOffset: -1
AlignAfterOpenBracket: Align
AlignConsecutiveMacros: false
AlignConsecutiveAssignments: false
AlignConsecutiveDeclarations: false
AlignEscapedNewlines: Left
AlignOperands: true
AlignTrailingComments: true
AllowAllArgumentsOnNextLine: true
AllowAllConstructorInitializersOnNextLine: true
AllowAllParametersOfDeclarationOnNextLine: true
AllowShortBlocksOnASingleLine: Never
AllowShortCaseLabelsOnASingleLine: false
AllowShortFunctionsOnASingleLine: All
AllowShortLambdasOnASingleLine: All
AllowShortIfStatementsOnASingleLine: WithoutElse
AllowShortLoopsOnASingleLine: true
AlwaysBreakAfterDefinitionReturnType: None
AlwaysBreakAfterReturnType: None
AlwaysBreakBeforeMultilineStrings: true
AlwaysBreakTemplateDeclarations: Yes
BinPackArguments: true
BinPackParameters: true
BraceWrapping:
AfterCaseLabel: false
AfterClass: false
AfterControlStatement: false
AfterEnum: false
AfterFunction: false
AfterNamespace: false
AfterObjCDeclaration: false
AfterStruct: false
AfterUnion: false
AfterExternBlock: false
BeforeCatch: false
BeforeElse: false
IndentBraces: false
SplitEmptyFunction: true
SplitEmptyRecord: true
SplitEmptyNamespace: true
BreakBeforeBinaryOperators: None
BreakBeforeBraces: Attach
BreakBeforeInheritanceComma: false
BreakInheritanceList: BeforeColon
BreakBeforeTernaryOperators: true
BreakConstructorInitializersBeforeComma: false
BreakConstructorInitializers: BeforeColon
BreakAfterJavaFieldAnnotations: false
BreakStringLiterals: true
ColumnLimit: 80
CommentPragmas: '^ IWYU pragma:'
CompactNamespaces: false
ConstructorInitializerAllOnOneLineOrOnePerLine: true
ConstructorInitializerIndentWidth: 4
ContinuationIndentWidth: 4
Cpp11BracedListStyle: true
DeriveLineEnding: true
DerivePointerAlignment: false
DisableFormat: false
ExperimentalAutoDetectBinPacking: false
FixNamespaceComments: true
ForEachMacros:
- foreach
- Q_FOREACH
- BOOST_FOREACH
IncludeBlocks: Regroup
IncludeCategories:
- Regex: '"postgres\.h"'
Priority: -1
- Regex: '^<gtest/gtest\.h>'
Priority: -1
- Regex: '"comm/cbdb_api\.h"'
Priority: 1
- Regex: '^<.*\.h>'
Priority: 2
SortPriority: 0
- Regex: '^<.*'
Priority: 3
SortPriority: 0
- Regex: '^<ext/.*\.h>'
Priority: 4
SortPriority: 0
- Regex: '.*'
Priority: 5
SortPriority: 0
IncludeIsMainRegex: '([-_](test|unittest))?$'
IncludeIsMainSourceRegex: ''
IndentCaseLabels: true
IndentGotoLabels: true
IndentPPDirectives: None
IndentWidth: 2
IndentWrappedFunctionNames: false
JavaScriptQuotes: Leave
JavaScriptWrapImports: true
KeepEmptyLinesAtTheStartOfBlocks: false
MacroBlockBegin: ''
MacroBlockEnd: ''
MaxEmptyLinesToKeep: 1
NamespaceIndentation: None
ObjCBinPackProtocolList: Never
ObjCBlockIndentWidth: 2
ObjCSpaceAfterProperty: false
ObjCSpaceBeforeProtocolList: true
PenaltyBreakAssignment: 2
PenaltyBreakBeforeFirstCallParameter: 1
PenaltyBreakComment: 300
PenaltyBreakFirstLessLess: 120
PenaltyBreakString: 1000
PenaltyBreakTemplateDeclaration: 10
PenaltyExcessCharacter: 1000000
PenaltyReturnTypeOnItsOwnLine: 200
PointerAlignment: Right
ReferenceAlignment: Pointer
RawStringFormats:
- Language: Cpp
Delimiters:
- cc
- CC
- cpp
- Cpp
- CPP
- 'c++'
- 'C++'
CanonicalDelimiter: ''
BasedOnStyle: google
- Language: TextProto
Delimiters:
- pb
- PB
- proto
- PROTO
EnclosingFunctions:
- EqualsProto
- EquivToProto
- PARSE_PARTIAL_TEXT_PROTO
- PARSE_TEST_PROTO
- PARSE_TEXT_PROTO
- ParseTextOrDie
- ParseTextProtoOrDie
CanonicalDelimiter: ''
BasedOnStyle: google
ReflowComments: true
SortIncludes: true
SortUsingDeclarations: true
SpaceAfterCStyleCast: false
SpaceAfterLogicalNot: false
SpaceAfterTemplateKeyword: true
SpaceBeforeAssignmentOperators: true
SpaceBeforeCpp11BracedList: false
SpaceBeforeCtorInitializerColon: true
SpaceBeforeInheritanceColon: true
SpaceBeforeParens: ControlStatements
SpaceBeforeRangeBasedForLoopColon: true
SpaceInEmptyBlock: false
SpaceInEmptyParentheses: false
SpacesBeforeTrailingComments: 2
SpacesInAngles: false
SpacesInConditionalStatement: false
SpacesInContainerLiterals: true
SpacesInCStyleCastParentheses: false
SpacesInParentheses: false
SpacesInSquareBrackets: false
SpaceBeforeSquareBrackets: false
Standard: Auto
StatementMacros:
- Q_UNUSED
- QT_REQUIRE_VERSION
TabWidth: 8
UseCRLF: false
UseTab: Never
...

Loading
Loading