SlideShare ist ein Scribd-Unternehmen logo
1 von 46
Downloaden Sie, um offline zu lesen
Writing A Foreign Data Wrapper
Bernd Helmle, bernd.helmle@credativ.de

30. Oktober 2013
Why FDWs?

...it is in the SQL Standard (SQL/MED)
...migration
...heterogeneous infrastructure
...integration of remote (non-)relational datasources
...it’s cool
http:
//rhaas.blogspot.com/2011/01/why-sqlmed-is-cool.html
Access remote datasources as PostgreSQL tables...

CREATE EXTENSION IF NOT EXISTS informix_fdw;
CREATE SERVER sles11_tcp FOREIGN DATA WRAPPER informix_fdw OPTIONS (
informixdir ’/Applications/IBM/informix’,
informixserver ’ol_informix1170’
);
CREATE USER MAPPING FOR bernd SERVER sles11_tcp OPTIONS (
password ’informix’,
username ’informix’
);
CREATE FOREIGN TABLE bar (
id integer,
value text
)
SERVER sles11_tcp
OPTIONS (
client_locale ’en_US.utf8’, database ’test’,
db_locale ’en_US.819’, query ’SELECT * FROM bar’
);
SELECT * FROM bar;
What we need...

...a C-interface to our remote datasource
...knowledge about PostgreSQL’s FDW API
...an idea how we deal with errors
...how remote data can be mapped to PostgreSQL datatypes
...time and steadiness
Python-Gurus also could use http://multicorn.org/.
Before you start your own...

Have a look at
http://wiki.postgresql.org/wiki/Foreign_data_wrappers
Let’s start...

extern Datum ifx_fdw_handler(PG_FUNCTION_ARGS);
extern Datum ifx_fdw_validator(PG_FUNCTION_ARGS);
CREATE FUNCTION ifx_fdw_handler() RETURNS fdw_handler
AS ’MODULE_PATHNAME’
LANGUAGE C STRICT;
CREATE FUNCTION ifx_fdw_validator(text[], oid) RETURNS void
AS ’MODULE_PATHNAME’
LANGUAGE C STRICT;
CREATE FOREIGN DATA WRAPPER informix_fdw
HANDLER ifx_fdw_handler
VALIDATOR ifx_fdw_validator;
FDW handler
Creates and initializes a FdwRoutine structure, example:
Datum
ifx_fdw_handler(PG_FUNCTION_ARGS)
{
FdwRoutine *fdwRoutine = makeNode(FdwRoutine);
fdwRoutine->ExplainForeignScan = ifxExplainForeignScan;
fdwRoutine->BeginForeignScan
= ifxBeginForeignScan;
fdwRoutine->IterateForeignScan = ifxIterateForeignScan;
fdwRoutine->EndForeignScan
= ifxEndForeignScan;
fdwRoutine->ReScanForeignScan = ifxReScanForeignScan;
#if PG_VERSION_NUM < 90200
fdwRoutine->PlanForeignScan

= ifxPlanForeignScan;

#else
fdwRoutine->GetForeignRelSize = ifxGetForeignRelSize;
fdwRoutine->GetForeignPaths
= ifxGetForeignPaths;
fdwRoutine->GetForeignPlan
= ifxGetForeignPlan;
#endif
FDW handler
...and much more callbacks for DML with PostgreSQL 9.3
Datum
ifx_fdw_handler(PG_FUNCTION_ARGS)
{
[...]
#if PG_VERSION_NUM >= 90300
fdwRoutine->AddForeignUpdateTargets
fdwRoutine->PlanForeignModify
fdwRoutine->BeginForeignModify
fdwRoutine->ExecForeignInsert
fdwRoutine->ExecForeignDelete
fdwRoutine->ExecForeignUpdate
fdwRoutine->EndForeignModify
fdwRoutine->IsForeignRelUpdatable
#endif
PG_RETURN_POINTER(fdwRoutine);
}

=
=
=
=
=
=
=
=

ifxAddForeignUpdateTargets;
ifxPlanForeignModify;
ifxBeginForeignModify;
ifxExecForeignInsert;
ifxExecForeignDelete;
ifxExecForeignUpdate;
ifxEndForeignModify;
ifxIsForeignRelUpdatable;
FDW validator callback
Called via CREATE FOREIGN TABLE or
ALTER FOREIGN TABLE
Validates a List * of FDW options.
Use untransformRelOptions() to get a list of FDW options
Don’t forget to test for duplicated options!
Up to you which options you want to support
Have a look into foreign/foreign.h for various helper
functions
FDW API callback routines (1)
#ifdef PG_VERSION_NUM < 90200
static FdwPlan *PlanForeignScan(Oid foreignTableOid,
PlannerInfo *planInfo,
RelOptInfo *baserel);
#else
static void GetForeignRelSize(PlannerInfo *root,
RelOptInfo *baserel,
Oid foreignTableId);
static void GetForeignPaths(PlannerInfo *root,
RelOptInfo *baserel,
Oid foreignTableId);
static ForeignScan *GetForeignPlan(PlannerInfo *root,
RelOptInfo *baserel,
Oid foreignTableId,
ForeignPath *best_path,
List *tlist,
List *scan_clauses);
#endif
FDW API callback routines (2)

static void ExplainForeignScan(ForeignScanState *node,
ExplainState *es);

static void BeginForeignScan(ForeignScanState *node, int eflags);

static TupleTableSlot *IterateForeignScan(ForeignScanState *node);

static void EndForeignScan(ForeignScanState *node);
FDW API callback routines (3)

9.2 has callbacks for ANALYZE, too:
bool
AnalyzeForeignTable (Relation relation,
AcquireSampleRowsFunc *func,
BlockNumber *totalpages);
int
AcquireSampleRowsFunc (Relation relation, int elevel,
HeapTuple *rows, int targrows,
double *totalrows,
double *totaldeadrows);
FDW API callback routines (4)

9.3 introduces callbacks for DML actions
#if PG_VERSION_NUM >= 90300
fdwRoutine->AddForeignUpdateTargets
fdwRoutine->PlanForeignModify
fdwRoutine->BeginForeignModify
fdwRoutine->ExecForeignInsert
fdwRoutine->ExecForeignDelete
fdwRoutine->ExecForeignUpdate
fdwRoutine->EndForeignModify
fdwRoutine->IsForeignRelUpdatable
fdwRoutine->ExplainForeignModify
#endif

=
=
=
=
=
=
=
=
=

ifxAddForeignUpdateTargets;
ifxPlanForeignModify;
ifxBeginForeignModify;
ifxExecForeignInsert;
ifxExecForeignDelete;
ifxExecForeignUpdate;
ifxEndForeignModify;
ifxIsForeignRelUpdatable;
ifxExplainForeignModify;
FDW Flow
AddForeignTargets()
GetForeignRelSize()
GetForeignPaths()

Query Cost Estimates
Query Planning
Query Planning - Finalize
Prepare / Metadata

GetForeignPlan()
PlanForeignModify()

IsForeignRelUpdatable()

BeginForeignScan()
BeginForeignModify()

Materialize

IterateForeignScan()
ExecForeignUpdate()
ExecForeignDelete()
ExecForeignInsert()

EndForeignScan()

Free resources
AddForeignTargets

static void
ifxAddForeignUpdateTargets(Query *parsetree,
RangeTblEntry *target_rte,
Relation target_relation);
Injects a custom column into the parsetree for DELETE and
UPDATE commands.
Can be NULL in case DELETE is not supported or UPDATE
actions rely on unchanged unique keys.
FDW Query Planning

Setup and Planning a scan or modify action on a foreign
datasource
E.g. establish and cache remote connection
Initialize required supporting structures for remote access
Planner info and cost estimates via baserel and root
parameters.
Big differences between 9.1 and 9.2 API, DML introduced
with 9.3 API
GetForeignRelSize() (1)

Size estimates for remote datasource (table size, number of
rows, ...)
root: Query Information Structure
baserel: Table Information Structure, carry your FDW
private information in baserel->fdw private.
baserel->fdw private is a pointer to your state data
structure.
GetForeignRelSize() (2)

baserel->rows
= ifxGetEstimatedNRows(&coninfo);
baserel->width
= ifxGetEstimatedRowSize(&coninfo);
baserel->fdw_private = (void *) planState;
GetForeignPaths() (1)

Create access path for foreign datasource.
ForeignPath access path required at least.
Multiple paths possible (e.g. presorted results, ...)
Arbitrarily complex
GetForeignPaths() (2)

planState = (IfxFdwPlanState *) baserel->fdw_private;
/*
* Create a generic foreign path for now. We need to consider any
* restriction quals later, to get a smarter path generation here.
*
* For example, it is quite interesting to consider any index scans
* or sorted output on the remote side and reflect it in the
* choosen paths (helps nested loops et al.).
*/
add_path(baserel, (Path *)
create_foreignscan_path(root, baserel,
baserel->rows,
planState->coninfo->planData.costs,
planState->coninfo->planData.costs,
NIL,
NULL,
NIL));
GetForeignPlan() (1)

Creates a final ForeignScan plan node based on paths
created by GetForeignPaths()
Additional parameters
ForeignPath *best path: Choosen foreign access path
(best)
List *tlist: Target list
List *scan clauses: Restriction clauses enforced by the
plan
GetForeignPlan() (2)

ForeignScan plan node should be created by
make foreignscan():
ForeignScan *
make_foreignscan(List *qptlist,
List *qpqual,
Index scanrelid,
List *fdw_exprs,
List *fdw_private)

fdw exprs: Expressions to be evaluated by the planner
fdw private: Private FDW data
Passing FDW planning info to execution state

Save parameters in the foreignScan->fdw_private pointer.
Must be copieable with copyObject.
Use a List * with either bytea or/and constant values (via
makeConst).
List *plan_values;

plan_values = NIL;
plan_values = lappend(plan_values,
makeConst(BYTEAOID, -1, InvalidOid, -1,
PointerGetDatum(ifxFdwPlanDataAsBytea(coninfo))
false, false));
Predicate Pushdown

Challenge: Filter the data on the remote dataset before transferring
them
Nobody wants to filter thousands of rows to just get one
Idea: push filter conditions down to the foreign datasource (if
possible)
Ideally done during planning phase (GetForeignRelSize(),
GetForeignPaths())
baserel->baserestrictinfo
Hard to get it right
Predicate Pushdown
baserel->baserestrictinfo: List of predicates belonging
to the foreign table (logically AND’ed)
baserel->reltargetlist: List of columns belonging to the
foreign table
Have a look at expression tree walker() and ruleutils API
(include/nodes/nodeFuncs.h,
include/utils/ruleutils.h)
ListCell *cell;
foreach(cell, baserel->baserestrictinfo)
{
RestrictInfo *info;
info = (RestrictInfo *) lfirst(cell);
if (IsA(info->clause, OpExpr))
{
/* examine right and left operand */
}
}
Predicate Pushdown, Example

SELECT COUNT(*) FROM sles11.inttest;
count
------10001
(1 row)
EXPLAIN SELECT * FROM foo JOIN centosifx.inttest t
ON (t.f1 = foo.id)
WHERE t.f1 = 104 AND t.f2 = 120;
QUERY PLAN
--------------------------------------------------------------------------------Nested Loop (cost=2.00..1695.03 rows=1 width=18)
-> Seq Scan on foo (cost=0.00..1693.01 rows=1 width=4)
Filter: (id = 104)
-> Foreign Scan on inttest t (cost=2.00..2.01 rows=1 width=14)
Filter: ((f1 = 104) AND (f2 = 120))
Informix costs: 2.00
Informix query: SELECT * FROM inttest WHERE (f1 = 104) AND (f2 = 120)
(7 rows)
PlanForeignModify

Create modify state information, e.g. get a connection,
describe remote metadata, ...
Private state information attached to ModifyTable plan node
State information pushed down to BeginForeignModify
Same with planning a scan: private state information must be
in a List * format.
INSERT probably needs more work here (connection
preparing, ...)
PlanForeignModify - Access scan state
Possible to access the scan state created during planning phase
static List *
ifxPlanForeignModify(PlannerInfo *root,
ModifyTable *plan,
Index resultRelation,
int subplan_index)
{
if ((resultRelation < root->simple_rel_array_size)
&& (root->simple_rel_array[resultRelation] != NULL))
{
RelOptInfo *relInfo = root->simple_rel_array[resultRelation];
IfxFdwExecutionState *scan_state;
/*
* Extract the state of the foreign scan.
*/
scan_state = (IfxFdwExecutionState *)
((IfxFdwPlanState *)relInfo->fdw_private)->state;
}
}
BeginForeignScan()

void
BeginForeignScan (ForeignScanState *node,
int eflags);
Execute startup callback for the FDW.
Basically prepares the FDW for executing a scan.
ForeignScanState saves function state values.
Use node->fdw_state to assign your own FDW state
structure.
Must handle EXPLAIN and EXPLAIN ANALYZE by checking
eflags & EXEC_FLAG_EXPLAIN_ONLY
BeginForeignModify

Like BeginForeignScan(), starting callback for DML actions
into the executor.
Depending on the DML action, might be differences in
preparing stuff
E.g. setup connection or get connection from cache
Deserialize plan data pushed down from
PlanForeignModify()
IterateForeignScan() (1)

TupleTableSlot *
IterateForeignScan (ForeignScanState *node);
Fetches data from the remote source.
Data conversion
Materializes a physical or virtual tuple to be returned.
Needs to return an empty tuple when done.
Private FDW data located in node->fdw state
IterateForeignScan() (2)

Returning a virtual tuple
TupleTableSlot *slot = node->ss.ss_ScanTupleSlot;
slot->tts_isempty = false;
slot->tts_nvalid = number_cols;;
slot->tts_values = (Datum *)palloc(sizeof(Datum) * slot->tts_nvalid);
slot->tts_isnull = (bool *)palloc(sizeof(bool) * slot->tts_nvalid);
for (i = 0; j < attrCount - 1; i)
{
tupleSlot->tts_isnull[i] = false;
tupleSlot->tts_values[i] = PointerGetDatum(val);
}
ReScanForeignScan()

void ReScanForeignScan (ForeignScanState *node);
Prepares the FDW to handle a rescan
Begins the scan from the beginning, e.g when used in
scrollable cursors
Must take care for changed query parameters!
Better to just “instruct” IterateForeignScan() to do the
right thing (tm)
Execute a modify action(1)

static TupleTableSlot *
ifxExecForeignUpdate(EState *estate,
ResultRelInfo *rinfo,
TupleTableSlot *slot,
TupleTableSlot *planSlot)
static TupleTableSlot *
ifxExecForeignInsert(EState *estate,
ResultRelInfo *rinfo,
TupleTableSlot *slot,
TupleTableSlot *planSlot);
static TupleTableSlot *
ifxExecForeignDelete(EState *estate,
ResultRelInfo *rinfo,
TupleTableSlot *slot,
TupleTableSlot *planSlot);
Execute a modify action (2)

ExecForeignUpdate, ExecForeignDelete and
ExecForeignInsert callbacks
Called for every tuple to work on
An INSERT action doesn’t employ a remote scan!
Does all the necessary data conversion if necessary
slot carries all information for the target tuple
Private FDW data stored in rinfo->ri FdwState
EndForeignScan(), EndForeignModify()

void EndForeignScan (ForeignScanState *node);
void EndForeignModify(EState *estate,
ResultRelInfo *rinfo)
EndForeignModify() run when no more tuple to modify
EndForeignScan() run when IterateForeignScan returns
no more rows, but after EndForeignModify()
Finalizes the remote scan and modify action
Close result sets, handles, connection, free memory, etc...
Transaction Callbacks

Connect local transactions with remote transactions and savepoints
RegisterXactCallback(pgfdw_xact_callback, NULL);
RegisterSubXactCallback(pgfdw_subxact_callback, NULL);
static void fdw_xact_callback(XactEvent event, void *arg);
static void fdw_subxact_callback(SubXactEvent event, SubTransactionId mySubid,
SubTransactionId parentSubid, void *arg)
Memory Management

PostgreSQL uses palloc()
Memory is allocated in CurrentMemoryContext
Use your own MemoryContext where necessary (e.g.
IterateForeignScan())
Memory allocated in external libraries need special care
Data conversion

Easy, if the remote datasource delivers a well formatted value
string (e.g. date strings formatted as yyyy-mm-dd).
Use type input function directly
Binary compatible types (e.g integer)
Binary data should always be bytea
String data must have a valid encoding!
Data conversion - Encoding

Within a FDW, a backend acts like any other client: ensure
encoding compatibility or encode your string data properly.
A look at mb/pg wchar.h might be of interest.
GetDatabaseEncoding()
pg_do_encoding_conversion()
Data conversion - Get type input function

regproc
typinputfunc;
Datum
result;
HeapTuple type_tuple;
type_tuple = SearchSysCache1(TYPEOID, inputOid);
if (!HeapTupleIsValid(type_tuple))
{
/*
* Oops, this is not expected...
*/
...
}
ReleaseSysCache(type_tuple);
typinputfunc = ((Form_pg_type) GETSTRUCT(type_tuple))->typinput;
result = OidFunctionCall2(typinputfunc,
CStringGetDatum(buf),
ObjectIdGetDatum(InvalidOid));
Error Handling (1)

Set FDW SQLSTATE according to your error condition class
HV, see http: // www. postgresql. org/ docs/ 9. 3/
static/ errcodes-appendix. html
Alternative: map remote error conditions to PostgreSQL errors
Be careful with elog(ERROR, ...).
Error Handling (2)

Example (there is no FDW WARNING SQLSTATE):
if (err == IFX_CONNECTION_WARN)
{
IfxSqlStateMessage message;
ifxGetSqlStateMessage(1, &message);
ereport(WARNING, (errcode(WARNING),
errmsg("opened informix connection with warnings"),
errdetail("informix SQLSTATE %s: "%s"",
message.sqlstate, message.text)));
}
Catching Errors (1)

Sometimes necessary to catch backend errors
Synchronize error conditions between PostgreSQL and remote
datasource
Possibility: use a PG TRY...PG CATCH block.
Catching Errors (2)

PG_TRY();
{
...
typinputfunc = getTypeInputFunction(state, PG_ATTRTYPE_P(state, attnum));
result = OidFunctionCall2(typinputfunc,
CStringGetDatum(pstrdup(buf)),
ObjectIdGetDatum(InvalidOid));
}
PG_CATCH();
{
ifxRewindCallstack(&(state->stmt_info));
PG_RE_THROW();
}
PG_END_TRY();
Thank You!

Más contenido relacionado

Was ist angesagt?

H cat berlinbuzzwords2012
H cat berlinbuzzwords2012H cat berlinbuzzwords2012
H cat berlinbuzzwords2012Hortonworks
 
OSDC 2012 | Scaling with MongoDB by Ross Lawley
OSDC 2012 | Scaling with MongoDB by Ross LawleyOSDC 2012 | Scaling with MongoDB by Ross Lawley
OSDC 2012 | Scaling with MongoDB by Ross LawleyNETWAYS
 
Zend Server Data Caching
Zend Server Data CachingZend Server Data Caching
Zend Server Data CachingEl Taller Web
 
Hive data migration (export/import)
Hive data migration (export/import)Hive data migration (export/import)
Hive data migration (export/import)Bopyo Hong
 
Hypertable
HypertableHypertable
Hypertablebetaisao
 
Database Architectures and Hypertable
Database Architectures and HypertableDatabase Architectures and Hypertable
Database Architectures and Hypertablehypertable
 
Hypertable - massively scalable nosql database
Hypertable - massively scalable nosql databaseHypertable - massively scalable nosql database
Hypertable - massively scalable nosql databasebigdatagurus_meetup
 
Tiered storage intro. By Robert Hodges, Altinity CEO
Tiered storage intro. By Robert Hodges, Altinity CEOTiered storage intro. By Robert Hodges, Altinity CEO
Tiered storage intro. By Robert Hodges, Altinity CEOAltinity Ltd
 
Hadoop 20111117
Hadoop 20111117Hadoop 20111117
Hadoop 20111117exsuns
 
July 2010 Triangle Hadoop Users Group - Chad Vawter Slides
July 2010 Triangle Hadoop Users Group - Chad Vawter SlidesJuly 2010 Triangle Hadoop Users Group - Chad Vawter Slides
July 2010 Triangle Hadoop Users Group - Chad Vawter Slidesryancox
 
Rapid Development of Data Generators Using Meta Generators in PDGF
Rapid Development of Data Generators Using Meta Generators in PDGFRapid Development of Data Generators Using Meta Generators in PDGF
Rapid Development of Data Generators Using Meta Generators in PDGFTilmann Rabl
 
Data Exploration with Apache Drill: Day 1
Data Exploration with Apache Drill:  Day 1Data Exploration with Apache Drill:  Day 1
Data Exploration with Apache Drill: Day 1Charles Givre
 
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오PgDay.Seoul
 
Replication and replica sets
Replication and replica setsReplication and replica sets
Replication and replica setsRandall Hunt
 

Was ist angesagt? (20)

H cat berlinbuzzwords2012
H cat berlinbuzzwords2012H cat berlinbuzzwords2012
H cat berlinbuzzwords2012
 
OSDC 2012 | Scaling with MongoDB by Ross Lawley
OSDC 2012 | Scaling with MongoDB by Ross LawleyOSDC 2012 | Scaling with MongoDB by Ross Lawley
OSDC 2012 | Scaling with MongoDB by Ross Lawley
 
Postgresql Federation
Postgresql FederationPostgresql Federation
Postgresql Federation
 
Intro To Couch Db
Intro To Couch DbIntro To Couch Db
Intro To Couch Db
 
Mongodb replication
Mongodb replicationMongodb replication
Mongodb replication
 
Zend Server Data Caching
Zend Server Data CachingZend Server Data Caching
Zend Server Data Caching
 
Hive data migration (export/import)
Hive data migration (export/import)Hive data migration (export/import)
Hive data migration (export/import)
 
Hypertable
HypertableHypertable
Hypertable
 
Database Architectures and Hypertable
Database Architectures and HypertableDatabase Architectures and Hypertable
Database Architectures and Hypertable
 
Hypertable - massively scalable nosql database
Hypertable - massively scalable nosql databaseHypertable - massively scalable nosql database
Hypertable - massively scalable nosql database
 
Tiered storage intro. By Robert Hodges, Altinity CEO
Tiered storage intro. By Robert Hodges, Altinity CEOTiered storage intro. By Robert Hodges, Altinity CEO
Tiered storage intro. By Robert Hodges, Altinity CEO
 
Hadoop 20111117
Hadoop 20111117Hadoop 20111117
Hadoop 20111117
 
July 2010 Triangle Hadoop Users Group - Chad Vawter Slides
July 2010 Triangle Hadoop Users Group - Chad Vawter SlidesJuly 2010 Triangle Hadoop Users Group - Chad Vawter Slides
July 2010 Triangle Hadoop Users Group - Chad Vawter Slides
 
Introduction to Hadoop
Introduction to HadoopIntroduction to Hadoop
Introduction to Hadoop
 
Rapid Development of Data Generators Using Meta Generators in PDGF
Rapid Development of Data Generators Using Meta Generators in PDGFRapid Development of Data Generators Using Meta Generators in PDGF
Rapid Development of Data Generators Using Meta Generators in PDGF
 
Data Exploration with Apache Drill: Day 1
Data Exploration with Apache Drill:  Day 1Data Exploration with Apache Drill:  Day 1
Data Exploration with Apache Drill: Day 1
 
Hive commands
Hive commandsHive commands
Hive commands
 
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
 
Replication and replica sets
Replication and replica setsReplication and replica sets
Replication and replica sets
 
Jvm goes big_data_sfjava
Jvm goes big_data_sfjavaJvm goes big_data_sfjava
Jvm goes big_data_sfjava
 

Andere mochten auch

Foreign Data Wrappers and You with Postgres
Foreign Data Wrappers and You with PostgresForeign Data Wrappers and You with Postgres
Foreign Data Wrappers and You with PostgresEDB
 
EnterpriseDB Positioned in Leaders Quadrant in Gartner Magic Quadrant
EnterpriseDB Positioned in Leaders Quadrant in Gartner Magic QuadrantEnterpriseDB Positioned in Leaders Quadrant in Gartner Magic Quadrant
EnterpriseDB Positioned in Leaders Quadrant in Gartner Magic QuadrantEDB
 
NoSQL on ACID: Meet Unstructured Postgres
NoSQL on ACID: Meet Unstructured PostgresNoSQL on ACID: Meet Unstructured Postgres
NoSQL on ACID: Meet Unstructured PostgresEDB
 
Postgres NoSQL - Delivering Apps Faster
Postgres NoSQL - Delivering Apps FasterPostgres NoSQL - Delivering Apps Faster
Postgres NoSQL - Delivering Apps FasterEDB
 
The Security-Enhanced PostgreSQL - "system wide" consistency in access control
The Security-Enhanced PostgreSQL - "system wide" consistency in access controlThe Security-Enhanced PostgreSQL - "system wide" consistency in access control
The Security-Enhanced PostgreSQL - "system wide" consistency in access controlKohei KaiGai
 
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres OpenKevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres OpenPostgresOpen
 
PGEncryption_Tutorial
PGEncryption_TutorialPGEncryption_Tutorial
PGEncryption_TutorialVibhor Kumar
 
PostgresOpen 2013 A Comparison of PostgreSQL Encryption Options
PostgresOpen 2013 A Comparison of PostgreSQL Encryption OptionsPostgresOpen 2013 A Comparison of PostgreSQL Encryption Options
PostgresOpen 2013 A Comparison of PostgreSQL Encryption OptionsFaisal Akber
 
Postgres.foreign.data.wrappers.2015
Postgres.foreign.data.wrappers.2015Postgres.foreign.data.wrappers.2015
Postgres.foreign.data.wrappers.2015EDB
 
The NoSQL Way in Postgres
The NoSQL Way in PostgresThe NoSQL Way in Postgres
The NoSQL Way in PostgresEDB
 
Achieving Pci Compliace
Achieving Pci CompliaceAchieving Pci Compliace
Achieving Pci CompliaceDenish Patel
 
Security Best Practices for your Postgres Deployment
Security Best Practices for your Postgres DeploymentSecurity Best Practices for your Postgres Deployment
Security Best Practices for your Postgres DeploymentPGConf APAC
 
Backup recovery with PostgreSQL
Backup recovery with PostgreSQLBackup recovery with PostgreSQL
Backup recovery with PostgreSQLFederico Campoli
 

Andere mochten auch (13)

Foreign Data Wrappers and You with Postgres
Foreign Data Wrappers and You with PostgresForeign Data Wrappers and You with Postgres
Foreign Data Wrappers and You with Postgres
 
EnterpriseDB Positioned in Leaders Quadrant in Gartner Magic Quadrant
EnterpriseDB Positioned in Leaders Quadrant in Gartner Magic QuadrantEnterpriseDB Positioned in Leaders Quadrant in Gartner Magic Quadrant
EnterpriseDB Positioned in Leaders Quadrant in Gartner Magic Quadrant
 
NoSQL on ACID: Meet Unstructured Postgres
NoSQL on ACID: Meet Unstructured PostgresNoSQL on ACID: Meet Unstructured Postgres
NoSQL on ACID: Meet Unstructured Postgres
 
Postgres NoSQL - Delivering Apps Faster
Postgres NoSQL - Delivering Apps FasterPostgres NoSQL - Delivering Apps Faster
Postgres NoSQL - Delivering Apps Faster
 
The Security-Enhanced PostgreSQL - "system wide" consistency in access control
The Security-Enhanced PostgreSQL - "system wide" consistency in access controlThe Security-Enhanced PostgreSQL - "system wide" consistency in access control
The Security-Enhanced PostgreSQL - "system wide" consistency in access control
 
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres OpenKevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
 
PGEncryption_Tutorial
PGEncryption_TutorialPGEncryption_Tutorial
PGEncryption_Tutorial
 
PostgresOpen 2013 A Comparison of PostgreSQL Encryption Options
PostgresOpen 2013 A Comparison of PostgreSQL Encryption OptionsPostgresOpen 2013 A Comparison of PostgreSQL Encryption Options
PostgresOpen 2013 A Comparison of PostgreSQL Encryption Options
 
Postgres.foreign.data.wrappers.2015
Postgres.foreign.data.wrappers.2015Postgres.foreign.data.wrappers.2015
Postgres.foreign.data.wrappers.2015
 
The NoSQL Way in Postgres
The NoSQL Way in PostgresThe NoSQL Way in Postgres
The NoSQL Way in Postgres
 
Achieving Pci Compliace
Achieving Pci CompliaceAchieving Pci Compliace
Achieving Pci Compliace
 
Security Best Practices for your Postgres Deployment
Security Best Practices for your Postgres DeploymentSecurity Best Practices for your Postgres Deployment
Security Best Practices for your Postgres Deployment
 
Backup recovery with PostgreSQL
Backup recovery with PostgreSQLBackup recovery with PostgreSQL
Backup recovery with PostgreSQL
 

Ähnlich wie Writing A Foreign Data Wrapper

Accessing external hadoop data sources using pivotal e xtension framework (px...
Accessing external hadoop data sources using pivotal e xtension framework (px...Accessing external hadoop data sources using pivotal e xtension framework (px...
Accessing external hadoop data sources using pivotal e xtension framework (px...Sameer Tiwari
 
Implementing a basic directory-tree structure that is derived from a.pdf
Implementing a basic directory-tree structure that is derived from a.pdfImplementing a basic directory-tree structure that is derived from a.pdf
Implementing a basic directory-tree structure that is derived from a.pdffunkybabyindia
 
Working with databases in Perl
Working with databases in PerlWorking with databases in Perl
Working with databases in PerlLaurent Dami
 
Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Masahiro Nagano
 
Zend Framework 1.9 Setup & Using Zend_Tool
Zend Framework 1.9 Setup & Using Zend_ToolZend Framework 1.9 Setup & Using Zend_Tool
Zend Framework 1.9 Setup & Using Zend_ToolGordon Forsythe
 
Trivadis TechEvent 2016 Polybase challenges Hive relational access to non-rel...
Trivadis TechEvent 2016 Polybase challenges Hive relational access to non-rel...Trivadis TechEvent 2016 Polybase challenges Hive relational access to non-rel...
Trivadis TechEvent 2016 Polybase challenges Hive relational access to non-rel...Trivadis
 
Getting Started with PL/Proxy
Getting Started with PL/ProxyGetting Started with PL/Proxy
Getting Started with PL/ProxyPeter Eisentraut
 
Extbase and Beyond
Extbase and BeyondExtbase and Beyond
Extbase and BeyondJochen Rau
 
Php Extensions for Dummies
Php Extensions for DummiesPhp Extensions for Dummies
Php Extensions for DummiesElizabeth Smith
 
Create your own PHP extension, step by step - phpDay 2012 Verona
Create your own PHP extension, step by step - phpDay 2012 VeronaCreate your own PHP extension, step by step - phpDay 2012 Verona
Create your own PHP extension, step by step - phpDay 2012 VeronaPatrick Allaert
 
Intro to Hadoop
Intro to HadoopIntro to Hadoop
Intro to Hadoopjeffturner
 

Ähnlich wie Writing A Foreign Data Wrapper (20)

Accessing external hadoop data sources using pivotal e xtension framework (px...
Accessing external hadoop data sources using pivotal e xtension framework (px...Accessing external hadoop data sources using pivotal e xtension framework (px...
Accessing external hadoop data sources using pivotal e xtension framework (px...
 
Hachiojipm11
Hachiojipm11Hachiojipm11
Hachiojipm11
 
Implementing a basic directory-tree structure that is derived from a.pdf
Implementing a basic directory-tree structure that is derived from a.pdfImplementing a basic directory-tree structure that is derived from a.pdf
Implementing a basic directory-tree structure that is derived from a.pdf
 
Working with databases in Perl
Working with databases in PerlWorking with databases in Perl
Working with databases in Perl
 
Quebec pdo
Quebec pdoQuebec pdo
Quebec pdo
 
Quebec pdo
Quebec pdoQuebec pdo
Quebec pdo
 
Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7Operation Oriented Web Applications / Yokohama pm7
Operation Oriented Web Applications / Yokohama pm7
 
Drupal 7 database api
Drupal 7 database api Drupal 7 database api
Drupal 7 database api
 
Sysprog17
Sysprog17Sysprog17
Sysprog17
 
Apache Hive
Apache HiveApache Hive
Apache Hive
 
Fatc
FatcFatc
Fatc
 
Sql user group
Sql user groupSql user group
Sql user group
 
02 20180605 meetup_fdw_v1
02 20180605 meetup_fdw_v102 20180605 meetup_fdw_v1
02 20180605 meetup_fdw_v1
 
Zend Framework 1.9 Setup & Using Zend_Tool
Zend Framework 1.9 Setup & Using Zend_ToolZend Framework 1.9 Setup & Using Zend_Tool
Zend Framework 1.9 Setup & Using Zend_Tool
 
Trivadis TechEvent 2016 Polybase challenges Hive relational access to non-rel...
Trivadis TechEvent 2016 Polybase challenges Hive relational access to non-rel...Trivadis TechEvent 2016 Polybase challenges Hive relational access to non-rel...
Trivadis TechEvent 2016 Polybase challenges Hive relational access to non-rel...
 
Getting Started with PL/Proxy
Getting Started with PL/ProxyGetting Started with PL/Proxy
Getting Started with PL/Proxy
 
Extbase and Beyond
Extbase and BeyondExtbase and Beyond
Extbase and Beyond
 
Php Extensions for Dummies
Php Extensions for DummiesPhp Extensions for Dummies
Php Extensions for Dummies
 
Create your own PHP extension, step by step - phpDay 2012 Verona
Create your own PHP extension, step by step - phpDay 2012 VeronaCreate your own PHP extension, step by step - phpDay 2012 Verona
Create your own PHP extension, step by step - phpDay 2012 Verona
 
Intro to Hadoop
Intro to HadoopIntro to Hadoop
Intro to Hadoop
 

Último

My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxSatishbabu Gunukula
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 

Último (20)

My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptx
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 

Writing A Foreign Data Wrapper

  • 1. Writing A Foreign Data Wrapper Bernd Helmle, bernd.helmle@credativ.de 30. Oktober 2013
  • 2. Why FDWs? ...it is in the SQL Standard (SQL/MED) ...migration ...heterogeneous infrastructure ...integration of remote (non-)relational datasources ...it’s cool http: //rhaas.blogspot.com/2011/01/why-sqlmed-is-cool.html
  • 3. Access remote datasources as PostgreSQL tables... CREATE EXTENSION IF NOT EXISTS informix_fdw; CREATE SERVER sles11_tcp FOREIGN DATA WRAPPER informix_fdw OPTIONS ( informixdir ’/Applications/IBM/informix’, informixserver ’ol_informix1170’ ); CREATE USER MAPPING FOR bernd SERVER sles11_tcp OPTIONS ( password ’informix’, username ’informix’ ); CREATE FOREIGN TABLE bar ( id integer, value text ) SERVER sles11_tcp OPTIONS ( client_locale ’en_US.utf8’, database ’test’, db_locale ’en_US.819’, query ’SELECT * FROM bar’ ); SELECT * FROM bar;
  • 4. What we need... ...a C-interface to our remote datasource ...knowledge about PostgreSQL’s FDW API ...an idea how we deal with errors ...how remote data can be mapped to PostgreSQL datatypes ...time and steadiness Python-Gurus also could use http://multicorn.org/.
  • 5. Before you start your own... Have a look at http://wiki.postgresql.org/wiki/Foreign_data_wrappers
  • 6. Let’s start... extern Datum ifx_fdw_handler(PG_FUNCTION_ARGS); extern Datum ifx_fdw_validator(PG_FUNCTION_ARGS); CREATE FUNCTION ifx_fdw_handler() RETURNS fdw_handler AS ’MODULE_PATHNAME’ LANGUAGE C STRICT; CREATE FUNCTION ifx_fdw_validator(text[], oid) RETURNS void AS ’MODULE_PATHNAME’ LANGUAGE C STRICT; CREATE FOREIGN DATA WRAPPER informix_fdw HANDLER ifx_fdw_handler VALIDATOR ifx_fdw_validator;
  • 7. FDW handler Creates and initializes a FdwRoutine structure, example: Datum ifx_fdw_handler(PG_FUNCTION_ARGS) { FdwRoutine *fdwRoutine = makeNode(FdwRoutine); fdwRoutine->ExplainForeignScan = ifxExplainForeignScan; fdwRoutine->BeginForeignScan = ifxBeginForeignScan; fdwRoutine->IterateForeignScan = ifxIterateForeignScan; fdwRoutine->EndForeignScan = ifxEndForeignScan; fdwRoutine->ReScanForeignScan = ifxReScanForeignScan; #if PG_VERSION_NUM < 90200 fdwRoutine->PlanForeignScan = ifxPlanForeignScan; #else fdwRoutine->GetForeignRelSize = ifxGetForeignRelSize; fdwRoutine->GetForeignPaths = ifxGetForeignPaths; fdwRoutine->GetForeignPlan = ifxGetForeignPlan; #endif
  • 8. FDW handler ...and much more callbacks for DML with PostgreSQL 9.3 Datum ifx_fdw_handler(PG_FUNCTION_ARGS) { [...] #if PG_VERSION_NUM >= 90300 fdwRoutine->AddForeignUpdateTargets fdwRoutine->PlanForeignModify fdwRoutine->BeginForeignModify fdwRoutine->ExecForeignInsert fdwRoutine->ExecForeignDelete fdwRoutine->ExecForeignUpdate fdwRoutine->EndForeignModify fdwRoutine->IsForeignRelUpdatable #endif PG_RETURN_POINTER(fdwRoutine); } = = = = = = = = ifxAddForeignUpdateTargets; ifxPlanForeignModify; ifxBeginForeignModify; ifxExecForeignInsert; ifxExecForeignDelete; ifxExecForeignUpdate; ifxEndForeignModify; ifxIsForeignRelUpdatable;
  • 9. FDW validator callback Called via CREATE FOREIGN TABLE or ALTER FOREIGN TABLE Validates a List * of FDW options. Use untransformRelOptions() to get a list of FDW options Don’t forget to test for duplicated options! Up to you which options you want to support Have a look into foreign/foreign.h for various helper functions
  • 10. FDW API callback routines (1) #ifdef PG_VERSION_NUM < 90200 static FdwPlan *PlanForeignScan(Oid foreignTableOid, PlannerInfo *planInfo, RelOptInfo *baserel); #else static void GetForeignRelSize(PlannerInfo *root, RelOptInfo *baserel, Oid foreignTableId); static void GetForeignPaths(PlannerInfo *root, RelOptInfo *baserel, Oid foreignTableId); static ForeignScan *GetForeignPlan(PlannerInfo *root, RelOptInfo *baserel, Oid foreignTableId, ForeignPath *best_path, List *tlist, List *scan_clauses); #endif
  • 11. FDW API callback routines (2) static void ExplainForeignScan(ForeignScanState *node, ExplainState *es); static void BeginForeignScan(ForeignScanState *node, int eflags); static TupleTableSlot *IterateForeignScan(ForeignScanState *node); static void EndForeignScan(ForeignScanState *node);
  • 12. FDW API callback routines (3) 9.2 has callbacks for ANALYZE, too: bool AnalyzeForeignTable (Relation relation, AcquireSampleRowsFunc *func, BlockNumber *totalpages); int AcquireSampleRowsFunc (Relation relation, int elevel, HeapTuple *rows, int targrows, double *totalrows, double *totaldeadrows);
  • 13. FDW API callback routines (4) 9.3 introduces callbacks for DML actions #if PG_VERSION_NUM >= 90300 fdwRoutine->AddForeignUpdateTargets fdwRoutine->PlanForeignModify fdwRoutine->BeginForeignModify fdwRoutine->ExecForeignInsert fdwRoutine->ExecForeignDelete fdwRoutine->ExecForeignUpdate fdwRoutine->EndForeignModify fdwRoutine->IsForeignRelUpdatable fdwRoutine->ExplainForeignModify #endif = = = = = = = = = ifxAddForeignUpdateTargets; ifxPlanForeignModify; ifxBeginForeignModify; ifxExecForeignInsert; ifxExecForeignDelete; ifxExecForeignUpdate; ifxEndForeignModify; ifxIsForeignRelUpdatable; ifxExplainForeignModify;
  • 14. FDW Flow AddForeignTargets() GetForeignRelSize() GetForeignPaths() Query Cost Estimates Query Planning Query Planning - Finalize Prepare / Metadata GetForeignPlan() PlanForeignModify() IsForeignRelUpdatable() BeginForeignScan() BeginForeignModify() Materialize IterateForeignScan() ExecForeignUpdate() ExecForeignDelete() ExecForeignInsert() EndForeignScan() Free resources
  • 15. AddForeignTargets static void ifxAddForeignUpdateTargets(Query *parsetree, RangeTblEntry *target_rte, Relation target_relation); Injects a custom column into the parsetree for DELETE and UPDATE commands. Can be NULL in case DELETE is not supported or UPDATE actions rely on unchanged unique keys.
  • 16. FDW Query Planning Setup and Planning a scan or modify action on a foreign datasource E.g. establish and cache remote connection Initialize required supporting structures for remote access Planner info and cost estimates via baserel and root parameters. Big differences between 9.1 and 9.2 API, DML introduced with 9.3 API
  • 17. GetForeignRelSize() (1) Size estimates for remote datasource (table size, number of rows, ...) root: Query Information Structure baserel: Table Information Structure, carry your FDW private information in baserel->fdw private. baserel->fdw private is a pointer to your state data structure.
  • 18. GetForeignRelSize() (2) baserel->rows = ifxGetEstimatedNRows(&coninfo); baserel->width = ifxGetEstimatedRowSize(&coninfo); baserel->fdw_private = (void *) planState;
  • 19. GetForeignPaths() (1) Create access path for foreign datasource. ForeignPath access path required at least. Multiple paths possible (e.g. presorted results, ...) Arbitrarily complex
  • 20. GetForeignPaths() (2) planState = (IfxFdwPlanState *) baserel->fdw_private; /* * Create a generic foreign path for now. We need to consider any * restriction quals later, to get a smarter path generation here. * * For example, it is quite interesting to consider any index scans * or sorted output on the remote side and reflect it in the * choosen paths (helps nested loops et al.). */ add_path(baserel, (Path *) create_foreignscan_path(root, baserel, baserel->rows, planState->coninfo->planData.costs, planState->coninfo->planData.costs, NIL, NULL, NIL));
  • 21. GetForeignPlan() (1) Creates a final ForeignScan plan node based on paths created by GetForeignPaths() Additional parameters ForeignPath *best path: Choosen foreign access path (best) List *tlist: Target list List *scan clauses: Restriction clauses enforced by the plan
  • 22. GetForeignPlan() (2) ForeignScan plan node should be created by make foreignscan(): ForeignScan * make_foreignscan(List *qptlist, List *qpqual, Index scanrelid, List *fdw_exprs, List *fdw_private) fdw exprs: Expressions to be evaluated by the planner fdw private: Private FDW data
  • 23. Passing FDW planning info to execution state Save parameters in the foreignScan->fdw_private pointer. Must be copieable with copyObject. Use a List * with either bytea or/and constant values (via makeConst). List *plan_values; plan_values = NIL; plan_values = lappend(plan_values, makeConst(BYTEAOID, -1, InvalidOid, -1, PointerGetDatum(ifxFdwPlanDataAsBytea(coninfo)) false, false));
  • 24. Predicate Pushdown Challenge: Filter the data on the remote dataset before transferring them Nobody wants to filter thousands of rows to just get one Idea: push filter conditions down to the foreign datasource (if possible) Ideally done during planning phase (GetForeignRelSize(), GetForeignPaths()) baserel->baserestrictinfo Hard to get it right
  • 25. Predicate Pushdown baserel->baserestrictinfo: List of predicates belonging to the foreign table (logically AND’ed) baserel->reltargetlist: List of columns belonging to the foreign table Have a look at expression tree walker() and ruleutils API (include/nodes/nodeFuncs.h, include/utils/ruleutils.h) ListCell *cell; foreach(cell, baserel->baserestrictinfo) { RestrictInfo *info; info = (RestrictInfo *) lfirst(cell); if (IsA(info->clause, OpExpr)) { /* examine right and left operand */ } }
  • 26. Predicate Pushdown, Example SELECT COUNT(*) FROM sles11.inttest; count ------10001 (1 row) EXPLAIN SELECT * FROM foo JOIN centosifx.inttest t ON (t.f1 = foo.id) WHERE t.f1 = 104 AND t.f2 = 120; QUERY PLAN --------------------------------------------------------------------------------Nested Loop (cost=2.00..1695.03 rows=1 width=18) -> Seq Scan on foo (cost=0.00..1693.01 rows=1 width=4) Filter: (id = 104) -> Foreign Scan on inttest t (cost=2.00..2.01 rows=1 width=14) Filter: ((f1 = 104) AND (f2 = 120)) Informix costs: 2.00 Informix query: SELECT * FROM inttest WHERE (f1 = 104) AND (f2 = 120) (7 rows)
  • 27. PlanForeignModify Create modify state information, e.g. get a connection, describe remote metadata, ... Private state information attached to ModifyTable plan node State information pushed down to BeginForeignModify Same with planning a scan: private state information must be in a List * format. INSERT probably needs more work here (connection preparing, ...)
  • 28. PlanForeignModify - Access scan state Possible to access the scan state created during planning phase static List * ifxPlanForeignModify(PlannerInfo *root, ModifyTable *plan, Index resultRelation, int subplan_index) { if ((resultRelation < root->simple_rel_array_size) && (root->simple_rel_array[resultRelation] != NULL)) { RelOptInfo *relInfo = root->simple_rel_array[resultRelation]; IfxFdwExecutionState *scan_state; /* * Extract the state of the foreign scan. */ scan_state = (IfxFdwExecutionState *) ((IfxFdwPlanState *)relInfo->fdw_private)->state; } }
  • 29. BeginForeignScan() void BeginForeignScan (ForeignScanState *node, int eflags); Execute startup callback for the FDW. Basically prepares the FDW for executing a scan. ForeignScanState saves function state values. Use node->fdw_state to assign your own FDW state structure. Must handle EXPLAIN and EXPLAIN ANALYZE by checking eflags & EXEC_FLAG_EXPLAIN_ONLY
  • 30. BeginForeignModify Like BeginForeignScan(), starting callback for DML actions into the executor. Depending on the DML action, might be differences in preparing stuff E.g. setup connection or get connection from cache Deserialize plan data pushed down from PlanForeignModify()
  • 31. IterateForeignScan() (1) TupleTableSlot * IterateForeignScan (ForeignScanState *node); Fetches data from the remote source. Data conversion Materializes a physical or virtual tuple to be returned. Needs to return an empty tuple when done. Private FDW data located in node->fdw state
  • 32. IterateForeignScan() (2) Returning a virtual tuple TupleTableSlot *slot = node->ss.ss_ScanTupleSlot; slot->tts_isempty = false; slot->tts_nvalid = number_cols;; slot->tts_values = (Datum *)palloc(sizeof(Datum) * slot->tts_nvalid); slot->tts_isnull = (bool *)palloc(sizeof(bool) * slot->tts_nvalid); for (i = 0; j < attrCount - 1; i) { tupleSlot->tts_isnull[i] = false; tupleSlot->tts_values[i] = PointerGetDatum(val); }
  • 33. ReScanForeignScan() void ReScanForeignScan (ForeignScanState *node); Prepares the FDW to handle a rescan Begins the scan from the beginning, e.g when used in scrollable cursors Must take care for changed query parameters! Better to just “instruct” IterateForeignScan() to do the right thing (tm)
  • 34. Execute a modify action(1) static TupleTableSlot * ifxExecForeignUpdate(EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot) static TupleTableSlot * ifxExecForeignInsert(EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot); static TupleTableSlot * ifxExecForeignDelete(EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot);
  • 35. Execute a modify action (2) ExecForeignUpdate, ExecForeignDelete and ExecForeignInsert callbacks Called for every tuple to work on An INSERT action doesn’t employ a remote scan! Does all the necessary data conversion if necessary slot carries all information for the target tuple Private FDW data stored in rinfo->ri FdwState
  • 36. EndForeignScan(), EndForeignModify() void EndForeignScan (ForeignScanState *node); void EndForeignModify(EState *estate, ResultRelInfo *rinfo) EndForeignModify() run when no more tuple to modify EndForeignScan() run when IterateForeignScan returns no more rows, but after EndForeignModify() Finalizes the remote scan and modify action Close result sets, handles, connection, free memory, etc...
  • 37. Transaction Callbacks Connect local transactions with remote transactions and savepoints RegisterXactCallback(pgfdw_xact_callback, NULL); RegisterSubXactCallback(pgfdw_subxact_callback, NULL); static void fdw_xact_callback(XactEvent event, void *arg); static void fdw_subxact_callback(SubXactEvent event, SubTransactionId mySubid, SubTransactionId parentSubid, void *arg)
  • 38. Memory Management PostgreSQL uses palloc() Memory is allocated in CurrentMemoryContext Use your own MemoryContext where necessary (e.g. IterateForeignScan()) Memory allocated in external libraries need special care
  • 39. Data conversion Easy, if the remote datasource delivers a well formatted value string (e.g. date strings formatted as yyyy-mm-dd). Use type input function directly Binary compatible types (e.g integer) Binary data should always be bytea String data must have a valid encoding!
  • 40. Data conversion - Encoding Within a FDW, a backend acts like any other client: ensure encoding compatibility or encode your string data properly. A look at mb/pg wchar.h might be of interest. GetDatabaseEncoding() pg_do_encoding_conversion()
  • 41. Data conversion - Get type input function regproc typinputfunc; Datum result; HeapTuple type_tuple; type_tuple = SearchSysCache1(TYPEOID, inputOid); if (!HeapTupleIsValid(type_tuple)) { /* * Oops, this is not expected... */ ... } ReleaseSysCache(type_tuple); typinputfunc = ((Form_pg_type) GETSTRUCT(type_tuple))->typinput; result = OidFunctionCall2(typinputfunc, CStringGetDatum(buf), ObjectIdGetDatum(InvalidOid));
  • 42. Error Handling (1) Set FDW SQLSTATE according to your error condition class HV, see http: // www. postgresql. org/ docs/ 9. 3/ static/ errcodes-appendix. html Alternative: map remote error conditions to PostgreSQL errors Be careful with elog(ERROR, ...).
  • 43. Error Handling (2) Example (there is no FDW WARNING SQLSTATE): if (err == IFX_CONNECTION_WARN) { IfxSqlStateMessage message; ifxGetSqlStateMessage(1, &message); ereport(WARNING, (errcode(WARNING), errmsg("opened informix connection with warnings"), errdetail("informix SQLSTATE %s: "%s"", message.sqlstate, message.text))); }
  • 44. Catching Errors (1) Sometimes necessary to catch backend errors Synchronize error conditions between PostgreSQL and remote datasource Possibility: use a PG TRY...PG CATCH block.
  • 45. Catching Errors (2) PG_TRY(); { ... typinputfunc = getTypeInputFunction(state, PG_ATTRTYPE_P(state, attnum)); result = OidFunctionCall2(typinputfunc, CStringGetDatum(pstrdup(buf)), ObjectIdGetDatum(InvalidOid)); } PG_CATCH(); { ifxRewindCallstack(&(state->stmt_info)); PG_RE_THROW(); } PG_END_TRY();