You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs - pxf init/sync support to master standby (#7540)
* docs - pxf init/sync support to master standby
* edits requested by david
* edits requested by francisco and oliver
* pxf sync from master TO standby or seg host
* identify sync run on master in pxf sync option description
Copy file name to clipboardExpand all lines: gpdb-doc/markdown/pxf/access_hdfs.html.md.erb
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -46,8 +46,8 @@ The PXF agent invokes the HDFS Java API to read the data and delivers it to the
46
46
47
47
Before working with Hadoop data using PXF, ensure that:
48
48
49
-
- You have configured and initialized PXF on your Greenplum Database segment hosts, and PXF is running on each host. See [Configuring PXF](instcfg_pxf.html) for additional information.
50
-
- You have configured the PXF Hadoop Connectors that you plan to use on each Greenplum Database segment host. Refer to [Configuring PXF Hadoop Connectors](client_instcfg.html) for instructions. If you plan to access JSON-formatted data stored in a Cloudera Hadoop cluster, PXF requires a Cloudera version 5.8 or later Hadoop distribution.
49
+
- You have configured and initialized PXF, and PXF is running on each Greenplum Database segment host. See [Configuring PXF](instcfg_pxf.html) for additional information.
50
+
- You have configured the PXF Hadoop Connectors that you plan to use. Refer to [Configuring PXF Hadoop Connectors](client_instcfg.html) for instructions. If you plan to access JSON-formatted data stored in a Cloudera Hadoop cluster, PXF requires a Cloudera version 5.8 or later Hadoop distribution.
51
51
- If user impersonation is enabled (the default), ensure that you have granted read (and write as appropriate) permission to the HDFS files and directories that will be accessed as external tables in Greenplum Database to each Greenplum Database user/role name that will access the HDFS files and directories. If user impersonation is not enabled, you must grant this permission to the `gpadmin` user.
52
52
- Time is synchronized between the Greenplum Database segment hosts and the external Hadoop systems.
Copy file name to clipboardExpand all lines: gpdb-doc/markdown/pxf/access_objstore.html.md.erb
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -27,8 +27,8 @@ PXF is installed with connectors to Azure Blob Storage, Azure Data Lake, Google
27
27
28
28
Before working with object store data using PXF, ensure that:
29
29
30
-
- You have configured and initialized PXF on your Greenplum Database segment hosts, and PXF is running on each host. See [Configuring PXF](instcfg_pxf.html) for additional information.
31
-
- You have configured the PXF Object Store Connectors that you plan to use on each Greenplum Database segment host. Refer to [Configuring Connectors to Azure, Google Cloud Storage, Minio, and S3 Object Stores](objstore_cfg.html) for instructions.
30
+
- You have configured and initialized PXF, and PXF is running on each Greenplum Database segment host. See [Configuring PXF](instcfg_pxf.html) for additional information.
31
+
- You have configured the PXF Object Store Connectors that you plan to use. Refer to [Configuring Connectors to Azure, Google Cloud Storage, Minio, and S3 Object Stores](objstore_cfg.html) for instructions.
32
32
- Time is synchronized between the Greenplum Database segment hosts and the external object store systems.
Copy file name to clipboardExpand all lines: gpdb-doc/markdown/pxf/cfginitstart_pxf.html.md.erb
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -27,7 +27,7 @@ PXF provides two management commands:
27
27
- `pxf cluster` - manage all PXF service instances in the Greenplum Database cluster
28
28
- `pxf` - manage the PXF service instance on a specific Greenplum Database host
29
29
30
-
The [`pxf cluster`](ref/pxf-cluster.html) command supports `init`, `start`, `stop`, and `sync` subcommands. When you run a `pxf cluster` subcommand on the Greenplum Database master host, you perform the operation on all segment hosts in the Greenplum Database cluster.
30
+
The [`pxf cluster`](ref/pxf-cluster.html) command supports `init`, `start`, `stop`, and `sync` subcommands. When you run a `pxf cluster` subcommand on the Greenplum Database master host, you perform the operation on all segment hosts in the Greenplum Database cluster. PXF also runs the `init` and `sync` commands on the standby master host.
31
31
32
32
The [`pxf`](ref/pxf.html) command supports `init`, `start`, `stop`, `restart`, and `status` operations. These operations run locally. That is, if you want to start or stop the PXF agent on a specific Greenplum Database segment host, you log in to the host and run the command.
33
33
@@ -54,7 +54,7 @@ The `pxf-env.sh` file exposes the following PXF runtime configuration parameters
54
54
| PXF_KEYTAB | The absolute path to the PXF service Kerberos principal keytab file. | $PXF_CONF/keytabs/pxf.service.keytab |
55
55
| PXF_PRINCIPAL | The PXF service Kerberos principal. | gpadmin/\_HOST@EXAMPLE.COM |
56
56
57
-
You must synchronize any changes that you make to `pxf-env.sh`, `pxf-log4j.properties`, or `pxf-profiles.xml` to each Greenplum Database segment host, and (re)start PXF on each host.
57
+
You must synchronize any changes that you make to `pxf-env.sh`, `pxf-log4j.properties`, or `pxf-profiles.xml` to the Greenplum Database cluster, and (re)start PXF on each segment host.
Copy file name to clipboardExpand all lines: gpdb-doc/markdown/pxf/client_instcfg.html.md.erb
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -10,13 +10,13 @@ PXF is compatible with Cloudera, Hortonworks Data Platform, MapR, and generic Ap
10
10
11
11
Configuring PXF Hadoop connectors involves copying configuration files from your Hadoop cluster to the Greenplum Database master host. If you are using the MapR Hadoop distribution, you must also copy certain JAR files to the master host. Before you configure the PXF Hadoop connectors, ensure that you can copy files from hosts in your Hadoop cluster to the Greenplum Database master.
12
12
13
-
In this procedure, you copy Hadoop configuration files to the `$PXF_CONF/servers/default` directory on the Greenplum Database master host. You may also copy libraries to `$PXF_CONF/lib` for MapR support. You then synchronize the PXF configuration on the master host to the segment hosts. (PXF creates the`$PXF_CONF/*` directories when you run `pxf cluster init`.)
13
+
In this procedure, you copy Hadoop configuration files to the `$PXF_CONF/servers/default` directory on the Greenplum Database master host. You may also copy libraries to `$PXF_CONF/lib` for MapR support. You then synchronize the PXF configuration on the master host to the standby master and segment hosts. (PXF creates the`$PXF_CONF/*` directories when you run `pxf cluster init`.)
14
14
15
15
**Note**: After you complete the configuration procedure, you will have configured the PXF default Hadoop server. End users need not provide a `SERVER` option in a `CREATE EXTERNAL TABLE` command when they access the default Hadoop server configuration.
16
16
17
17
## <aid="client-pxf-config-steps"></a>Procedure
18
18
19
-
Perform the following procedure to configure the desired PXF Hadoop-related connectors on the Greenplum Database master host. After you configure the connectors, you will use the `pxf cluster sync` command to copy the PXF configuration to the segment hosts in your Greenplum Database cluster.
19
+
Perform the following procedure to configure the desired PXF Hadoop-related connectors on the Greenplum Database master host. After you configure the connectors, you will use the `pxf cluster sync` command to copy the PXF configuration to the Greenplum Database cluster.
20
20
21
21
1. Log in to your Greenplum Database master node:
22
22
@@ -55,7 +55,7 @@ Perform the following procedure to configure the desired PXF Hadoop-related conn
Copy file name to clipboardExpand all lines: gpdb-doc/markdown/pxf/init_pxf.html.md.erb
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -56,13 +56,13 @@ Perform the following procedure to initialize PXF on each segment host in your G
56
56
$ ssh gpadmin@<gpmaster>
57
57
```
58
58
59
-
4. Run the `pxf cluster init` command to initialize the PXF service on the master and on each segment host. For example, the following command specifies `/usr/local/greenplum-pxf` as the PXF user configuration directory for initialization:
59
+
4. Run the `pxf cluster init` command to initialize the PXF service on the master, standby master, and on each segment host. For example, the following command specifies `/usr/local/greenplum-pxf` as the PXF user configuration directory for initialization:
The `init` command creates the PXF web application and initializes the internal PXF configuration. The `init` command also creates the `$PXF_CONF` user configuration directory if it does not exist, and populates the directory with user-customizable configuration templates.
66
66
67
-
**Note**: The PXF service runs only on the segment hosts. However,`pxf cluster init` also sets up the PXF user configuration directories on the Greenplum Database master host.
67
+
**Note**: The PXF service runs only on the segment hosts. However,`pxf cluster init` also sets up the PXF user configuration directories on the Greenplum Database master and standby master hosts.
Copy file name to clipboardExpand all lines: gpdb-doc/markdown/pxf/install_java.html.md.erb
+7-6Lines changed: 7 additions & 6 deletions
Original file line number
Diff line number
Diff line change
@@ -13,35 +13,36 @@ Ensure that you have access to, or superuser permissions to install, Java versio
13
13
14
14
## <aid="proc"></a>Procedure
15
15
16
-
Perform the following procedure to install Java on the master and on each segment host in your Greenplum Database cluster. You will use the `gpssh` utility where possible to run a command on multiple hosts.
16
+
Perform the following procedure to install Java on the master, standby master, and on each segment host in your Greenplum Database cluster. You will use the `gpssh` utility where possible to run a command on multiple hosts.
17
17
18
18
1. Log in to your Greenplum Database master node:
19
19
20
20
``` shell
21
21
$ ssh gpadmin@<gpmaster>
22
22
```
23
23
24
-
2. Create a text file that lists your Greenplum Database segment hosts, one host name per line. For example, a file named `seghostfile` may include:
24
+
2. Create a text file that lists your Greenplum Database standby master host and segment hosts, one host name per line. For example, a file named `gphostfile` may include:
25
25
26
26
``` pre
27
+
mstandby
27
28
seghost1
28
29
seghost2
29
30
seghost3
30
31
```
31
32
32
-
3. Install Java on the master and on each Greenplum Database segment host, and then set up the Java environment on each host.
33
+
3. Install Java on the master, standby master, and on each Greenplum Database segment host, and then set up the Java environment on each host.
33
34
34
35
1. Install the Java package. For example, to install Java version 1.8:
2. Identify the Java base install directory. Update the `gpadmin` user's `.bashrc` file on each segment host to include this `$JAVA_HOME` setting if it is not already present. For example:
42
+
2. Identify the Java base install directory. Update the `gpadmin` user's `.bashrc` file on each host to include this `$JAVA_HOME` setting if it is not already present. For example:
Copy file name to clipboardExpand all lines: gpdb-doc/markdown/pxf/jdbc_cfg.html.md.erb
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -42,7 +42,7 @@ When you configure the PXF JDBC Connector to access an external SQL database, yo
42
42
2. Create the directory `$PXF_CONF/servers/<server_name>`.
43
43
3. Copy the PXF `jdbc-site.xml` template configuration file to the new server directory.
44
44
4. Fill in appropriate values for the properties in the template file.
45
-
6. Synchronize the server configuration to each Greenplum Database segment host.
45
+
6. Synchronize the server configuration to the Greenplum Database cluster.
46
46
7. Publish the PXF server name(s) to your Greenplum Database end users as appropriate.
47
47
48
48
The Greenplum Database user specifies the `<server_name>` in the `CREATE EXTERNAL TABLE` `LOCATION` clause `SERVER` option to access the external SQL database. For example, if you created a server configuration and named the server directory `pgsrvcfg`:
@@ -61,7 +61,7 @@ While not recommended, you can override a JDBC server configuration by directly
61
61
62
62
Ensure that you have initialized PXF before you configure a JDBC Connector server.
63
63
64
-
In this procedure, you name and add a PXF JDBC server configuration for a PostgreSQL database and synchronize the server configuration(s) to all segment hosts.
64
+
In this procedure, you name and add a PXF JDBC server configuration for a PostgreSQL database and synchronize the server configuration(s) to the Greenplum Database cluster.
65
65
66
66
1. Log in to your Greenplum Database master node:
67
67
@@ -110,7 +110,7 @@ In this procedure, you name and add a PXF JDBC server configuration for a Postgr
110
110
```
111
111
6. Save your changes and exit the editor.
112
112
113
-
7. Use the `pxf cluster sync` command to copy the new server configurations to each Greenplum Database segment host. For example:
113
+
7. Use the `pxf cluster sync` command to copy the new server configuration to the Greenplum Database cluster. For example:
Copy file name to clipboardExpand all lines: gpdb-doc/markdown/pxf/jdbc_pxf.html.md.erb
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -31,7 +31,7 @@ This section describes how to use the PXF JDBC connector to access data in an ex
31
31
32
32
Before you access an external SQL database using the PXF JDBC connector, ensure that:
33
33
34
-
- You have configured and initialized PXF on your Greenplum Database segment hosts, and PXF is running on each host. See [Configuring PXF](instcfg_pxf.html) for additional information.
34
+
- You have configured and initialized PXF, and PXF is running on each Greenplum Database segment host. See [Configuring PXF](instcfg_pxf.html) for additional information.
35
35
- You can identify the PXF user configuration directory (`$PXF_CONF`).
36
36
- Connectivity exists between all Greenplum Database segment hosts and the external SQL database.
37
37
- You have configured your external SQL database for user access from all Greenplum Database segment hosts.
@@ -222,7 +222,7 @@ Perform the following steps to create a PostgreSQL table named `forpxf_table1` i
222
222
223
223
#### <aid="ex_jdbconfig"></a>Configure the JDBC Connector
224
224
225
-
You must create a JDBC server configuration for PostgreSQL, download the PostgreSQL driver JAR file to your system, copy the JAR file to the PXF user configuration directory, synchronize PXF configuration, and then restart PXF.
225
+
You must create a JDBC server configuration for PostgreSQL, download the PostgreSQL driver JAR file to your system, copy the JAR file to the PXF user configuration directory, synchronize the PXF configuration, and then restart PXF.
226
226
227
227
1. Log in to the Greenplum Database master node:
228
228
@@ -262,7 +262,7 @@ You must create a JDBC server configuration for PostgreSQL, download the Postgre
Copy file name to clipboardExpand all lines: gpdb-doc/markdown/pxf/objstore_cfg.html.md.erb
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -46,7 +46,7 @@ When you configure a PXF object store connector, you add at least one named PXF
46
46
3. Copy the PXF template configuration file corresponding to the object store to the new server directory.
47
47
4. Fill in appropriate values for the properties in the template file.
48
48
5. Add additional properties and values if required for your environment.
49
-
6. Synchronize the server configuration to each Greenplum Database segment host.
49
+
6. Synchronize the server configuration to the Greenplum Database cluster.
50
50
7. Publish the PXF server names to your Greenplum Database end users as appropriate.
51
51
52
52
The Greenplum Database user specifies the server name in the `CREATE EXTERNAL TABLE` `LOCATION` clause `SERVER` option to access the object store. For example:
@@ -224,7 +224,7 @@ To enable SSE-C for a specific S3 bucket, use the property name variants that in
224
224
225
225
Ensure that you have initialized PXF before you configure an object store connector.
226
226
227
-
In this procedure, you name and add a PXF server configuration in the `$PXF_CONF/servers` directory on the Greenplum Database master host for each object store connector that you plan to use. You then use the `pxf cluster sync` command to sync the server configuration(s) to all segment hosts.
227
+
In this procedure, you name and add a PXF server configuration in the `$PXF_CONF/servers` directory on the Greenplum Database master host for each object store connector that you plan to use. You then use the `pxf cluster sync` command to sync the server configuration(s) to the Greenplum Database cluster.
228
228
229
229
1. Log in to your Greenplum Database master node:
230
230
@@ -275,7 +275,7 @@ In this procedure, you name and add a PXF server configuration in the `$PXF_CONF
275
275
276
276
6. Repeat Step 3 to configure the next object store connector.
277
277
278
-
4. Use the `pxf cluster sync` command to copy the new server configurations to each Greenplum Database segment host. For example:
278
+
4. Use the `pxf cluster sync` command to copy the new server configurations to the Greenplum Database cluster. For example:
0 commit comments