About Connector Options
You provide Greenplum Database connection options and the read/write options required by the Greenplum-Spark Connector via generic key-value String
pairs.
Greenplum-Spark Connector Options
The Greenplum-Spark Connector supports the options identified in the table below. An option is required unless otherwise specified.
Option Key | Value Description | Operation |
---|---|---|
url | The JDBC connection string URL; see Constructing the Greenplum Database JDBC URL. | Read, Write |
dbschema | The name of the Greenplum Database schema in which dbtable resides. This option also identifies the name of the schema in which the Greenplum-Spark Connector creates temporary external tables. Optional, the default schema is the schema named public . |
Read, Write |
dbtable | The name of the Greenplum Database table. When reading from Greenplum Database, this table or view must reside in the Greenplum Database schema identified in the dbschema option value. When writing from Spark to Greenplum Database, the Greenplum-Spark Connector creates the table in dbschema if the table does not already exist. |
Read, Write |
driver | The fully qualified class path of the custom JDBC driver. Optional, specify only when using a custom JDBC driver. | Read, Write |
user | The Greenplum Database user/role name. | Read, Write |
password | The Greenplum Database password for the user. You can omit the password if Greenplum Database is configured to not require a password for the specified user, or if you use kerberos authentication and provide the required authentication properties in the JDBC connection string URL. Optional. | Read, Write |
partitionColumn | The name of the Greenplum Database table column to use for Spark partitioning. This column must be one of the Greenplum Database data types integer , bigint , serial , or bigserial . The column name must be lowercase. |
Read |
partitions | The number of Spark partitions. Optional, the default value is the number of primary segments in the Greenplum Database cluster. | Read |
partitionsPerSegment | The number of Spark partitions per Greenplum Database segment. This option is ignored in favor of partitions . Note: The partitionsPerSegment option is deprecated as of version 1.5.0 and will be removed in a future release. |
Read |
truncate | The table overwrite mode. Governs the Connector’s table creation actions when you specify SaveMode.Overwrite on a write operation and the target Greenplum Database table exists. The default value is false ; the Connector drops and then re-creates the target table before it writes any data. When true , the Connector truncates the target table before writing any data. Optional. |
Write |
distributedBy | The distribution column(s) of the Greenplum table. Governs the table creation action of the Connector when the target Greenplum Database table does not exist, or when you specify SaveMode.Overwrite on a write operation and truncate is false . The Connector (re)creates the table with random distribution by default. When you provide one or more distributedBy columns, the Connector (re)creates the table with a DISTRIBUTED BY clause that specifies these column names. Optional. |
Write |
iteratorOptimization | The write memory optimization mode. Optional. The default value is true ; the Connector uses an Iterator to optimize memory. When false , the Connector materializes the data set in memory during the write operation. |
Write |
connector.port | The gpfdist server process port number, or the environment variable name that identifies such, on the Spark worker node. This option is ignored in favor of server.port . Note: The connector.port option is deprecated as of version 1.6.0 and will be removed in a future release. |
Read, Write |
server.port | The gpfdist server process port number, or the environment variable name that identifies such, on the Spark worker node. Optional, by default the Connector selects and uses a random port. |
Read, Write |
server.useHostname | Use the Spark worker node host name to specify the gpfdist server address. Optional, the default value is false . |
Read, Write |
server.hostEnv | The name of the environment variable whose value identifies the Spark worker node hostname or IP address on which to start the gpfdist server process. Optional. |
Read, Write |
server.nic | The name of the Spark worker node network interface on which to start the gpfdist server, or the environment variable name that identifies such. Optional. |
Read, Write |
pool.maxSize | The maximum number of connections in the connection pool. Optional, the default value is 64. | Read, Write |
pool.timeoutMs | The amount of time, in milliseconds, after which an inactive connection is considered idle. Optional, the default value is 10,000 (10 seconds). | Read, Write |
pool.minIdle | The minimum number of idle connections maintained in the connection pool. Optional, the default value is 0. | Read, Write |
You can specify options to the Connector individually or in an options Map
. The class methods of interest for setting Greenplum-Spark Connector options are:
.option(key: String, value: String)
for specifying an individual option, and:
.options(options: Map[String, String])
for specifying an options map.
To specify an option individually, provide <option_key> and <value> strings to the DataFrameReader.option()
method. For example, to specify the user
option:
.option("user", "gpdb_role_name")
To construct a scala.collection.Map
comprising more than one option, you provide the <option_key> and <value> strings for each option. For example, to create the options for a read operation:
val gscReadOptionMap = Map(
"url" -> "jdbc:postgresql://gpdb-master:5432/testdb",
"user" -> "bill",
"password" -> "changeme",
"dbschema" -> "myschema",
"dbtable" -> "table1",
"partitionColumn" -> "id"
)
To provide an options map to the Connector, specify it in the DataFrameReader.options()
or DataFrameWriter.options()
methods. For example, to provide the gscReadOptionMap
map created above:
.options(gscReadOptionMap)
Specifying Partition Options
Partition options apply only when you read data from Greenplum Database into Spark.
A Spark DataFrame
is a fault-tolerant collection of elements partitioned across the Spark cluster nodes. Spark operates on these elements in parallel.
Greenplum Database distributes its table data across segments running on segment hosts.
The Greenplum-Spark Connector provides two options to configure the mapping between Spark partitions and Greenplum Database segment data, partitionColumn
and partitions
.
partitionColumn
The partitionColumn
option that you specify must have the integer
, bigint
, serial
, or bigserial
Greenplum Database data type. The partitionColumn
you identify need not be the column you may have specified with the DISTRIBUTED BY (<column>)
clause when you created the Greenplum Database table.
partitions
By default, the Greenplum-Spark Connector creates one Spark partition per Greenplum Database segment. You can set the partitions
option to specify a larger number of Spark partitions.
Specifying the Connector Server Address Options
The Greenplum-Spark Connector uses the Greenplum Database gpfdist
protocol to transfer data between Spark and Greenplum Database.
Use one or more of the following options to specify the address and port number of the gpfdist
server running on a Spark worker node:
Refer to Configuring the Connector Server Address for additional information about these options.
server.port
You can specify one, a comma-separated list, and/or a range of port numbers, or an environment variable name for the server.port
option. To set a port number, identify the specific port number.
For example, to set server.port
to a single port number as a single option:
.option("server.port", "12900")
To set server.port
to include a range of port numbers within a Map
of options:
"server.port" -> "8999-9001, 9004"
When you specify more than one port number, the Greenplum-Spark Connector traverses the list or range in order, and uses the first available port number.
If you choose to specify the port number with an environment variable, prefix the environment variable name with env.
. For example, to identify the environment variable named GSC_EXTERNAL_PORT
as the server.port
option value:
"server.port" -> "env.GSC_EXTERNAL_PORT",
Note: Setting server.port
to env.GPFDIST_PORT
results in the same behavior as that of Greenplum-Spark Connector version 1.2.0.
server.useHostname
server.useHostname
is a boolean option that identifies whether to use the Spark worker host name or the IP address to identify the gpfdist
server on that node. When set to true
, the Greenplum-Spark Connector uses the Spark worker host name as the address of the gpfdist
server.
The default server.useHostname
value is false
. When false
and neither server.hostEnv
nor server.nic
are set (the default), the Greenplum-Spark Connector uses the Spark worker IP address to address the gpfdist
server.
To set server.useHostname
as a single option:
"server.useHostname" -> "true"
server.hostEnv
server.hostEnv
specifies the name of the environment variable whose value identifies the Spark worker node host name or IP address on which to start the gpfdist
server. This environment variable will typically identify the same information as the Spark SPARK_LOCAL_IP
environment variable.
The Connector examines the server.hostEnv
option only when server.useHostname
is set to false
.
To set server.hostEnv
as a single option:
"server.hostEnv" -> "env.MY_LOCAL_SPARK_IP"
server.nic
server.nic
identifies the name of the Spark worker node network interface on which to start the gpfdist
server, or the environment variable name that identifies such.
The Connector examines this option only when server.useHostname
is false
and no server.hostEnv
is specified.
To directly set server.nic
within a Map
of options:
"server.nic" -> "eth1"
Specifying Connection Pool Options
The Greenplum-Spark Connector provides connection pool configuration options. These options are named with the pool.
prefix:
Option Key | Value Description |
---|---|
pool.maxSize | The maximum number of connections in the connection pool. Optional, the default value is 64. |
pool.timeoutMs | The amount of time, in milliseconds, after which an inactive connection is considered idle. Optional, the default value is 10,000 (10 seconds). |
pool.minIdle | The minimum number of idle connections maintained in the connection pool. Optional, the default value is 0. |
To set each connection pool option as a single option:
.option("pool.maxSize", "50")
.option("pool.minIdle", "5")
.option("pool.timeoutMs", "7500")
The first DataFrame
that you create with a specific connection string URL, username, and password combination defines the configuration of that connection pool. The Greenplum-Spark Connector ignores connection pool options specified on subsequent DataFrame
s created with the same URL/username/password combination.
Refer to JDBC Connection Pooling for additional information about connection pool configuration options.