Pivotal Greenplum-Spark Connector 1.6.0 Release Notes
The Pivotal Greenplum-Spark Connector supports high speed, parallel data transfer between Greenplum Database and an Apache Spark cluster using:
- Spark’s Scala API - programmatic access (including the
Pivotal Greenplum-Spark Connector 1.6.0 is a minor release of the Greenplum Database connector for Apache Spark. This release includes new and changed features and bug fixes.
The following table identifies the supported component versions for Pivotal Greenplum-Spark Connector 1.6.0:
|Greenplum-Spark Connector Version||Greenplum Version||Spark Version||Scala Version|
|1.6.0||4.3.x, 5.x||2.1.2 and above||2.11|
|1.5.0||4.3.x, 5.x||2.1.2 and above||2.11|
Refer to the Pivotal Greenplum Database documentation for detailed information about Pivotal Greenplum Database.
See the Apache Spark documentation for information about Apache Spark version 2.1.2.
Pivotal Greenplum-Spark Connector 1.6.0 includes the following new feature:
Finer-Grained Control Over the Connector Server Address
The Greenplum-Spark Connector exposes new options to specify the
gpfdistserver process address on the Spark worker node. Refer to Configuring the Connector Server Address for additional information about these options.
Pivotal Greenplum-Spark Connector 1.6.0 includes the following changes:
connector.portOption is Replaced and Deprecated
The Greenplum-Spark Connector no longer uses the
connector.portoption. The Connector now uses an option named
server.portto identify the server port number.
The following issues were resolved in Pivotal Greenplum-Spark Connector version 1.6.0:
|29589||A read operation using the Greenplum-Spark Connector failed when the hosts in the Spark cluster were configured with multiple network interfaces. Greenplum Database was unable to access a
|29606||Due to a suboptimal table metadata query, the Greenplum-Spark Connector failed to read from a Greenplum Database view that contained greater than ten thousand rows. This issue is resolved. The Connector now uses a different query to obtain Greenplum table metadata.|
Known issues and limitations related to the 1.6.0 release of the Pivotal Greenplum-Spark Connector include the following:
- The Greenplum-Spark Connector supports basic data types like Float, Integer, String, and Date/Time data types. The Connector does not yet support more complex types. See Greenplum Database <-> Spark Data Type Mapping for additional information.