Skip to content

Commit 694e4e9

Browse files
tuhaihetuhaihe
committed
Remove bundled Python packages
This change removes the unnecessary bundled Python packages for a clear codebase and better ASF license compliance. Changes made: - Removed the `pythonSrc` and `pythonSrc/ext` directories from the source code. - Updated the `Makefile`, `configure` and other related files to remove the *pythonsrc_ext* related references. Instead building the Python packages like psutil, pygresql and pyyaml via the source code when running the Cloudberry Demo cluster, we can install them via the command under the top dir: ``` pip3 install -r requirements.txt ```
1 parent 3cd8f57 commit 694e4e9

43 files changed

Lines changed: 27 additions & 12394 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/coverity.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,6 @@ jobs:
107107
--with-perl \
108108
--with-pgport=5432 \
109109
--with-python \
110-
--with-pythonsrc-ext \
111110
--with-ssl=openssl \
112111
--with-uuid=e2fs \
113112
--with-includes=/usr/local/xerces-c/include \

‎configure‎

Lines changed: 6 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -570,6 +570,7 @@ ac_hostname=`(hostname || uname -n) 2>/dev/null | sed 1q`
570570
#
571571
# Initializations.
572572
#
573+
ac_default_prefix=/usr/local
573574
ac_clean_files=
574575
ac_config_libobj_dir=.
575576
LIBOBJS=
@@ -742,7 +743,6 @@ with_ldap
742743
with_krb_srvnam
743744
krb_srvtab
744745
with_gssapi
745-
with_pythonsrc_ext
746746
with_python
747747
with_perl
748748
with_tcl
@@ -758,8 +758,8 @@ HAVE_CXX14
758758
enable_gpcloud
759759
enable_shared_postgres_backend
760760
enable_mapreduce
761-
enable_catalog_ext
762761
enable_serverless
762+
enable_catalog_ext
763763
enable_orca
764764
autodepend
765765
PKG_CONFIG_LIBDIR
@@ -907,7 +907,6 @@ with_tcl
907907
with_tclconfig
908908
with_perl
909909
with_python
910-
with_pythonsrc_ext
911910
with_gssapi
912911
with_krb_srvnam
913912
with_pam
@@ -1606,7 +1605,7 @@ Optional Features:
16061605
--enable-cassert enable assertion checks (for debugging)
16071606
--disable-orca disable ORCA optimizer
16081607
--enable-catalog-ext enable Cloudberry catalog extension
1609-
--enable-serverless enable Cloudberry serverless mode
1608+
--enable-serverless use serverless mode of Cloudberry
16101609
--enable-mapreduce enable Cloudberry Mapreduce support
16111610
--disable-shared-postgres-backend
16121611
enable Cloudberry shared postgres backend support
@@ -1644,7 +1643,6 @@ Optional Packages:
16441643
--with-tclconfig=DIR tclConfig.sh is in DIR
16451644
--with-perl build Perl modules (PL/Perl)
16461645
--without-python build Python modules (PL/Python)
1647-
--with-pythonsrc-ext build Python modules for gpMgmt
16481646
--with-gssapi build with GSSAPI support
16491647
--with-krb-srvnam=NAME default service principal name in Kerberos (GSSAPI)
16501648
[postgres]
@@ -8377,7 +8375,7 @@ $as_echo "checking whether to build with catalog extension... $enable_catalog_ex
83778375

83788376

83798377
#
8380-
# Enable serverless mode
8378+
# --enable-serverless uses serverless mode of Cloudberry
83818379
#
83828380

83838381

@@ -8404,8 +8402,8 @@ else
84048402
fi
84058403

84068404

8407-
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: checking whether to use serverless architecture of Cloudberry ... $enable_serverless" >&5
8408-
$as_echo "checking whether to use serverless architecture of Cloudberry ... $enable_serverless" >&6; }
8405+
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: checking whether to use serverless mode of Cloudberry... $enable_serverless" >&5
8406+
$as_echo "checking whether to use serverless mode of Cloudberry... $enable_serverless" >&6; }
84098407

84108408

84118409
#
@@ -9423,40 +9421,6 @@ fi
94239421
$as_echo "$with_python" >&6; }
94249422

94259423

9426-
#
9427-
# Optionally build Python modules for gpMgmt
9428-
#
9429-
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build Python modules for gpMgmt" >&5
9430-
$as_echo_n "checking whether to build Python modules for gpMgmt... " >&6; }
9431-
9432-
9433-
9434-
# Check whether --with-pythonsrc-ext was given.
9435-
if test "${with_pythonsrc_ext+set}" = set; then :
9436-
withval=$with_pythonsrc_ext;
9437-
case $withval in
9438-
yes)
9439-
:
9440-
;;
9441-
no)
9442-
:
9443-
;;
9444-
*)
9445-
as_fn_error $? "no argument expected for --with-pythonsrc-ext option" "$LINENO" 5
9446-
;;
9447-
esac
9448-
9449-
else
9450-
with_pythonsrc_ext=no
9451-
9452-
fi
9453-
9454-
9455-
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_pythonsrc_ext" >&5
9456-
$as_echo "$with_pythonsrc_ext" >&6; }
9457-
9458-
9459-
94609424
#
94619425
# GSSAPI
94629426
#

‎configure.ac‎

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1011,15 +1011,6 @@ PGAC_ARG_BOOL(with, python, yes, [build Python modules (PL/Python)])
10111011
AC_MSG_RESULT([$with_python])
10121012
AC_SUBST(with_python)
10131013

1014-
#
1015-
# Optionally build Python modules for gpMgmt
1016-
#
1017-
AC_MSG_CHECKING([whether to build Python modules for gpMgmt])
1018-
PGAC_ARG_BOOL(with, pythonsrc-ext, no, [build Python modules for gpMgmt])
1019-
AC_MSG_RESULT([$with_pythonsrc_ext])
1020-
AC_SUBST(with_pythonsrc_ext)
1021-
1022-
10231014
#
10241015
# GSSAPI
10251016
#

‎gpAux/Makefile‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -124,13 +124,13 @@ DEFPORT=5432
124124

125125
ORCA_CONFIG=--enable-orca
126126

127-
rhel6_x86_64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap --with-pythonsrc-ext
128-
rhel7_x86_64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap --with-pythonsrc-ext
129-
rhel8_x86_64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap --with-pythonsrc-ext
130-
ubuntu18.04_x86_64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap --with-pythonsrc-ext
131-
sles12_x86_64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap --with-pythonsrc-ext
132-
rhel7_aarch64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap --with-pythonsrc-ext
133-
common_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap --with-pythonsrc-ext
127+
rhel6_x86_64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap
128+
rhel7_x86_64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap
129+
rhel8_x86_64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap
130+
ubuntu18.04_x86_64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap
131+
sles12_x86_64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap
132+
rhel7_aarch64_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap
133+
common_CONFIGFLAGS=--with-quicklz --with-lz4 --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --enable-gpcloud --with-libxml --with-ssl=openssl --with-pam --with-ldap
134134
#BLD_CONFIGFLAGS=$($(BLD_ARCH)_CONFIGFLAGS)
135135
BLD_CONFIGFLAGS=$(common_CONFIGFLAGS)
136136

‎gpMgmt/Makefile‎

Lines changed: 3 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -13,22 +13,9 @@ generate_greenplum_path_file:
1313

1414
install: generate_greenplum_path_file
1515
mkdir -p $(DESTDIR)$(prefix)/lib/python
16-
17-
# Setup /lib/python contents
18-
if [ -e bin/ext/__init__.py ]; then \
19-
cp -rp bin/ext/__init__.py $(DESTDIR)$(prefix)/lib/python ; \
20-
fi
21-
if [ -e bin/ext/psutil ]; then \
22-
cp -rp bin/ext/psutil $(DESTDIR)$(prefix)/lib/python ; \
23-
fi
24-
if [ -e bin/ext/pgdb.py ]; then \
25-
cp -rp bin/ext/pgdb.py $(DESTDIR)$(prefix)/lib/python && \
26-
cp -rp bin/ext/pg.py $(DESTDIR)$(prefix)/lib/python && \
27-
cp -rp bin/ext/_pg*.so $(DESTDIR)$(prefix)/lib/python ; \
28-
fi
29-
if [ -e bin/ext/yaml ]; then \
30-
cp -rp bin/ext/yaml $(DESTDIR)$(prefix)/lib/python ; \
31-
fi
16+
17+
# Remove PyGreSQL, psutil and PyYAML specific copy operations
18+
# These will be installed via pip
3219

3320
clean distclean:
3421
$(MAKE) -C bin $@

‎gpMgmt/bin/Makefile‎

Lines changed: 0 additions & 122 deletions
Original file line numberDiff line numberDiff line change
@@ -54,13 +54,10 @@ uninstall:
5454
# SOURCE DIRECTORIES
5555
#
5656
SRC=$(CURDIR)
57-
PYLIB_SRC=$(SRC)/pythonSrc
58-
PYLIB_SRC_EXT=$(PYLIB_SRC)/ext
5957
SBIN_DIR=$(SRC)/../sbin
6058
SERVER_SRC=$(SRC)
6159
SERVER_SBIN=$(SERVER_SRC)/../sbin
6260

63-
6461
#
6562
# INSTALL DIRECTORY
6663
#
@@ -70,128 +67,9 @@ PYLIB_DIR=$(SRC)/ext
7067
core:
7168
python3 gpconfig_modules/parse_guc_metadata.py $(DESTDIR)$(prefix)
7269

73-
ifeq ($(with_pythonsrc_ext), yes)
74-
install: installdirs installprograms core psutil pygresql pyyaml
75-
else
7670
install: installdirs installprograms core
77-
endif
78-
79-
#
80-
# Python Libraries
81-
#
82-
83-
#
84-
# PyGreSQL
85-
#
86-
PYGRESQL_VERSION=5.2
87-
PYGRESQL_DIR=PyGreSQL-$(PYGRESQL_VERSION)
88-
pygresql:
89-
@echo "--- PyGreSQL"
90-
cd $(PYLIB_SRC_EXT)/ && $(TAR) xzf $(PYGRESQL_DIR).tar.gz
91-
cd $(PYLIB_SRC_EXT)/$(PYGRESQL_DIR)/ && PATH=$(DESTDIR)$(bindir):$$PATH LDFLAGS='$(LDFLAGS) $(PYGRESQL_LDFLAGS)' python3 setup.py build
92-
cp -r $(PYLIB_SRC_EXT)/$(PYGRESQL_DIR)/build/lib*-3*/* $(PYLIB_DIR)/
93-
94-
95-
#
96-
# PSUTIL
97-
#
98-
PSUTIL_VERSION=5.7.0
99-
PSUTIL_DIR=psutil-$(PSUTIL_VERSION)
100-
101-
psutil:
102-
@echo "--- psutil"
103-
ifeq "$(findstring $(BLD_ARCH),aix7_ppc_64 )" ""
104-
cd $(PYLIB_SRC_EXT)/ && $(TAR) xzf $(PSUTIL_DIR).tar.gz
105-
cd $(PYLIB_SRC_EXT)/$(PSUTIL_DIR)/ && env -u CC python3 setup.py build
106-
cp -r $(PYLIB_SRC_EXT)/$(PSUTIL_DIR)/build/lib.*/psutil $(PYLIB_DIR)
107-
endif
108-
109-
#
110-
# PYYAML
111-
#
112-
PYYAML_VERSION=5.3.1
113-
PYYAML_DIR=PyYAML-$(PYYAML_VERSION)
114-
115-
pyyaml:
116-
@echo "--- pyyaml"
117-
cd $(PYLIB_SRC_EXT)/ && $(TAR) xzf $(PYYAML_DIR).tar.gz
118-
cd $(PYLIB_SRC_EXT)/$(PYYAML_DIR)/ && env -u CC python3 setup.py build
119-
cp -r $(PYLIB_SRC_EXT)/$(PYYAML_DIR)/build/lib*-3*/* $(PYLIB_DIR)/
120-
121-
#
122-
# PYLINT
123-
#
124-
125-
PYLINT_VERSION=0.21.0
126-
PYLINT_DIR=pylint-$(PYLINT_VERSION)
127-
PYLINT_PYTHONPATH=$(PYLIB_DIR):$(PYLIB_SRC_EXT)/$(PYLINT_DIR)/build/lib/
128-
MOCK_VERSION=1.0.1
129-
MOCK_DIR=mock-$(MOCK_VERSION)
130-
SETUP_TOOLS_VERSION=36.6.0
131-
PARSE_VERSION=1.8.2
132-
SETUP_TOOLS_DIR=setuptools-$(SETUP_TOOLS_VERSION)
133-
PARSE_DIR=parse-$(PARSE_VERSION)
134-
PYTHONSRC_INSTALL=$(PYLIB_SRC_EXT)/install
135-
PYTHON_VERSION=$(shell python3 -c "import sys; print ('%s.%s' % (sys.version_info[0:2]))")
136-
PYTHONSRC_INSTALL_SITE=$(PYLIB_SRC_EXT)/install/lib/python$(PYTHON_VERSION)/site-packages
137-
PYTHONSRC_INSTALL_PYTHON_PATH=$(PYTHONPATH):$(PYTHONSRC_INSTALL_SITE)
138-
# TODO: mock-1.0.1-py2.6.egg package should be updated.
139-
MOCK_BIN=$(PYTHONSRC_INSTALL)/lib/python$(PYTHON_VERSION)/site-packages/mock-1.0.1-py2.6.egg
140-
UBUNTU_PLATFORM=$(shell if lsb_release -a 2>/dev/null | grep -q 'Ubuntu' ; then echo "Ubuntu"; fi)
141-
142-
pylint:
143-
@echo "--- pylint"
144-
@cd $(PYLIB_SRC_EXT)/ && $(TAR) xzf $(PYLINT_DIR).tar.gz
145-
@cd $(PYLIB_SRC_EXT)/$(PYLINT_DIR)/ && python3 setup.py build 1> /dev/null
146-
@touch $(PYLIB_SRC_EXT)/$(PYLINT_DIR)/build/lib/__init__.py
147-
148-
$(MOCK_BIN):
149-
@echo "--- mock for platform $(UBUNTU_PLATFORM)"
150-
@if [ "$(UBUNTU_PLATFORM)" = "Ubuntu" ]; then\
151-
pip3 install mock;\
152-
else\
153-
mkdir -p $(PYTHONSRC_INSTALL_SITE) && \
154-
cd $(PYLIB_SRC_EXT)/ && $(TAR) xzf $(MOCK_DIR).tar.gz && \
155-
cd $(PYLIB_SRC_EXT)/$(MOCK_DIR)/ && \
156-
PYTHONPATH=$(PYTHONSRC_INSTALL_PYTHON_PATH) python3 setup.py install --prefix $(PYTHONSRC_INSTALL) ; \
157-
fi;
158-
159-
PYTHON_FILES=`grep -l --exclude=Makefile --exclude=gplogfilter "/bin/env python3" *`\
160-
`grep -l "/bin/env python3" $(SRC)/../sbin/*`\
161-
`find ./gppylib -name "*.py"`\
162-
`find $(SRC)/../sbin -name "*.py"`
163-
164-
checkcode: pylint
165-
@echo "Running pylint on management scripts..."
166-
@PYTHONPATH=$(PYTHONPATH):$(PYLINT_PYTHONPATH) $(PYLIB_SRC_EXT)/$(PYLINT_DIR)/bin/pylint -i y $(PYTHON_FILES) --rcfile=.rcfile > $(SRC)/../pylint.txt || true
167-
@echo -n "pylint_score=" > $(SRC)/../pylint_score.properties
168-
@grep "Your code has been rated at" $(SRC)/../pylint.txt | sed -e "s|Your .* \(.*\)/.*|\1|" >> $(SRC)/../pylint_score.properties
169-
170-
check: $(MOCK_BIN)
171-
@echo "Running pure unit and also "unit" tests that require cluster to be up..."
172-
@TMPDIR=/tmp PYTHONPATH=$(SERVER_SRC):$(SERVER_SBIN):$(PYTHONPATH):$(PYTHONSRC_INSTALL_PYTHON_PATH):$(SRC)/ext:$(SBIN_DIR):$(LIB_DIR):$(PYLIB_DIR)/mock-1.0.1 \
173-
gppylib/gpunit discover --verbose -s $(SRC)/gppylib -p "test_unit*.py" 2> $(SRC)/../gpMgmt_testunit_results.log 1> $(SRC)/../gpMgmt_testunit_output.log
174-
@TMPDIR=/tmp PYTHONPATH=$(SERVER_SRC):$(SERVER_SBIN):$(PYTHONPATH):$(PYTHONSRC_INSTALL_PYTHON_PATH):$(SRC)/ext:$(SBIN_DIR):$(LIB_DIR):$(PYLIB_DIR)/mock-1.0.1 \
175-
gppylib/gpunit discover --verbose -s $(SRC)/gppylib -p "test_cluster*.py" 2>> $(SRC)/../gpMgmt_testunit_results.log 1>> $(SRC)/../gpMgmt_testunit_output.log
176-
177-
unitdevel:
178-
@echo "Running pure unit tests..."
179-
PYTHONPATH=$(SERVER_SRC):$(SERVER_SBIN):$(PYTHONPATH):$(PYTHONSRC_INSTALL_PYTHON_PATH):$(SRC)/ext:$(SBIN_DIR):$(LIB_DIR):$(PYLIB_DIR)/mock-1.0.1 \
180-
python3 -m unittest discover --verbose -s $(SRC)/gppylib -p "test_unit*.py"
181-
182-
183-
.PHONY: installcheck-bash
184-
installcheck-bash:
185-
./test/suite.bash
186-
187-
.PHONY: installcheck
188-
installcheck: installcheck-bash
189-
$(MAKE) -C gpload_test $@
19071

19172
clean distclean:
192-
rm -rf $(PYLIB_SRC_EXT)/$(PYLINT_DIR)
193-
rm -rf $(PYLIB_SRC_EXT)/$(PYGRESQL_DIR)/build
194-
rm -rf $(PYLIB_SRC)/$(PYGRESQL_DIR)/build
19573
rm -rf *.pyc
19674
rm -f analyzedbc gpactivatestandbyc gpaddmirrorsc gpcheckcatc \
19775
gpcheckperfc gpcheckresgroupimplc gpchecksubnetcfgc gpconfigc \

‎gpMgmt/bin/README.md‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,19 @@
11
# For Developers
22

3-
To install the libraries necessary for running scripts or testing, a system python of 2.7 must be available, the version of gcc and g++ used to compile python must be available.
4-
On most distributions, python will compiled with the same gcc and g++ verion available from the base packages "gcc" and "gcc-c++".
3+
To install the libraries necessary for running scripts or testing, a system python of 3.x must be available.
54

6-
The command `python -VV` will show the compiler used to compile the version of python being used.
7-
A `make` in from gpMgmt will install the proper libraries provided a gcc and gcc-c++ are present.
5+
## Installing Python Dependencies
86

9-
To run any of these python scripts, necessary libraries must be installed, and PYTHONPATH must be modified to use the libraries in this path.
7+
Install required Python packages using pip:
108

9+
```bash
10+
pip3 install -r python-dependencies.txt
1111
```
12-
PYTHONPATH="\$GPHOME/lib/python:${PYTHONPATH}"
12+
13+
To run any of these python scripts, necessary libraries must be installed, and PYTHONPATH must be modified to use the libraries in this path.
14+
15+
```bash
16+
PYTHONPATH="$GPHOME/lib/python:${PYTHONPATH}"
1317
```
1418

1519
This will be set automatically with a `source $GPHOME/greenplum_path.sh`

‎gpMgmt/bin/gpload.bat‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
set FOUND=
33
for %%X in (python.exe) do (set FOUND=%%~$PATH:X)
44
if not defined FOUND (
5-
echo "Ensure python2.7 64-bit is installed and set in PATH"
5+
echo "Ensure python3 is installed and set in PATH"
66
exit /B 1
77
)
88

‎gpMgmt/bin/pythonSrc/PyGreSQL-4.0/docs/announce.html‎

Lines changed: 0 additions & 28 deletions
This file was deleted.

0 commit comments

Comments
 (0)