LATEST VERSION: 1.6 - RELEASE NOTES
Pivotal Greenplum®-Spark® Connector v1.6

Greenplum Database <-> Spark Data Type Mapping

When you load Greenplum data into Spark, or write Spark data into Greenplum, the Greenplum-Spark Connector maps Greenplum Database and Spark data types for you.

Note: The Greenplum-Spark Connector does not support complex types, nor does it support any data type not listed in the tables below.

For additional information on Greenplum Database or Spark data types, refer to:

Greenplum to Spark

The mapping from Greenplum Database data type to Spark data type (read) is identified in the table below:

Greenplum Data Type Spark Data Type
bigint LongType
bigSerial LongType
boolean BooleanType
char StringType
date DateType
decimal DecimalType
float4 FloatType
float8 DoubleType
int IntegerType
serial IntegerType
smallInt ShortType
text StringType
time TimeStampType
timestamp TimeStampType
timestamptz TimeStampType
timetz TimeStampType
varchar StringType

Spark to Greenplum

The mapping from Spark data type to Greenplum Database data type (write) is identified in the table below:

Spark Data Type Greenplum Data Type
BinaryType bytea
BooleanType boolean
DateType date
DecimalType numeric
DoubleType float8
FloatType float4
IntegerType int
LongType bigint
ShortType smallInt
StringType text
TimeStampType timestamp