2. Is This Your Current Relational Database Access
Style?
# Django ORM
>>> from ip2country.models import Ip2Country
>>> Ip2Country.objects.all()
[<Ip2Country: Ip2Country object>, <Ip2Country: Ip2Country
object>, '...(remaining elements truncated)...']
>>> myp = Ip2Country.objects.filter(assigned__year=2015)
... .filter(countrycode2=’MY')
>>> myp[0].ipfrom
736425984.0
3. Is This Your Current Relational Database Access
Style?
# SQLAlchemy ORM
>>> from sqlalchemy import create_engine, extract
>>> from sqlalchemy.orm import sessionmaker
>>> from models import Ip2Country
>>> engine =
create_engine('postgresql://ip2country_rw:secret@localhost/i
p2country')
>>> Session = sessionmaker(bind=engine)
>>> session = Session()
>>> all_data = session.query(Ip2Country).all()
>>> myp = session.query(Ip2Country).
... filter(extract('year',Ip2Country.assigned) == 2015).
... filter(Ip2Country.countrycode2 == ’MY')
print(myp[0].ipfrom)
736425984.0
4. SQL Relational Database Access
SELECT * FROM ip2country;
“id”,"ipfrom";"ipto";"registry";"assigned";"countrycode2";"countrycode3";"countryname"
1,1729522688;1729523711;"apnic";"2011-08-05";"CN";"CHN";"China"
2,1729523712;1729524735;"apnic";"2011-08-05";"CN";"CHN";"China”
. . .
SELECT * FROM ip2country
WHERE date_part('year', assigned) = 2015
AND countrycode2 = ’MY';
“id”,"ipfrom";"ipto";"registry";"assigned";"countrycode2";"countrycode3";"countryname"
5217;736425984;736427007;"apnic";"2015-01-13 00:00:00";"MY";"MYS";"Malaysia”
5218;736427008;736428031;"apnic";"2015-01-13 00:00:00";"MY";"MYS";"Malaysia”
. . .
SELECT ipfrom FROM ip2country
WHERE date_part('year', assigned) = 2015
AND countrycode2 = ’MY';
"ipfrom"
736425984
736427008
. . .
5. Python + SQL == Python DB-API 2.0
• The Python standard for a consistent
interface to relational databases is the
Python DB-API (PEP 249)
• The majority of Python database interfaces
adhere to this standard
7. Python DB-API Connection Object
Access the database via the connection object
• Use connect constructor to create a
connection with database
conn = psycopg2.connect(parameters…)
• Create cursor via the connection
cur = conn.cursor()
• Transaction management (implicit begin)
conn.commit()
conn.rollback()
• Close connection (will rollback current
transaction)
conn.close()
• Check module capabilities by globals
psycopg2.apilevel psycopg2.threadsafety
psycopg2.paramstyle
8. Python DB-API Cursor Object
A cursor object is used to represent a database
cursor, which is used to manage the context of
fetch operations.
• Cursors created from the same connection
are not isolated
cur = conn.cursor()
cur2 = conn.cursor()
• Cursor methods
cur.execute(operation, parameters)
cur.executemany(op,seq_of_parameters)
cur.fetchone()
cur.fetchmany([size=cursor.arraysize])
cur.fetchall()
cur.close()
9. Python DB-API Cursor Object
• Optional cursor methods
cur.scroll(value[,mode='relative'])
cur.next()
cur.callproc(procname[,parameters])
cur.__iter__()
• Results of an operation
cur.description
cur.rowcount
cur.lastrowid
• DB adaptor specific “proprietary” cursor
methods
10. Python DB-API Parameter Styles
Allows you to keep SQL separate from parameters
Improves performance & security
Warning Never, never, NEVER use Python string
concatenation (+) or string parameters
interpolation (%) to pass variables to a SQL query
string. Not even at gunpoint.
From http://initd.org/psycopg/docs/usage.html#query-parameters
11. Python DB-API Parameter Styles
Global paramstyle gives supported style for the
adaptor
qmark Question mark style
WHERE countrycode2 = ?
numeric Numeric positional style
WHERE countrycode2 = :1
named Named style
WHERE countrycode2 = :code
format ANSI C printf format style
WHERE countrycode2 = %s
pyformat Python format style
WHERE countrycode2 = %(name)s
12. Python + SQL: INSERT
import csv, datetime, psycopg2
conn = psycopg2.connect("dbname=ip2countrydb user=ip2country_rw
password=secret")
cur = conn.cursor()
with open("IpToCountry.csv", "rt") as f:
reader = csv.reader(f)
try:
for row in reader:
if row[0][0] != "#":
row[3] =
datetime.datetime.utcfromtimestamp(float(row[3]))
cur.execute("""INSERT INTO ip2country(
ipfrom, ipto, registry, assigned,
countrycode2, countrycode3, countryname)
VALUES (%s, %s, %s, %s, %s, %s, %s)""", row)
except (Exception) as error:
print(error)
conn.rollback()
else:
conn.commit()
finally:
cur.close()
conn.close()
13. Python + SQL: SELECT
# Find ipv4 address ranges assigned to Malaysia
import psycopg2, socket, struct
def num_to_dotted_quad(n):
"""convert long int to dotted quad string
http://code.activestate.com/recipes/66517/"""
return socket.inet_ntoa(struct.pack('!L', n))
conn = psycopg2.connect("dbname=ip2countrydb user=ip2country_rw
password=secret")
cur = conn.cursor()
cur.execute("""SELECT * FROM ip2country
WHERE countrycode2 = 'MY'
ORDER BY ipfrom""")
for row in cur:
print("%s - %s" % (num_to_dotted_quad(int(row[0])),
num_to_dotted_quad(int(row[1]))))
20. ODBC
• mxODBC
• CPython 2.3+
• DB-API 2.0 interfaces
• http://www.egenix.com/products/pytho
n/mxODBC/doc
• Commercial product
• PyODBC
• CPython 2 & 3
• DB-API 2.0 interfaces with extensions
• https://github.com/mkleehammer/pyod
bc
• ODBC interfaces not limited to Windows
thanks to iODBC and unixODBC
21. Jython + SQL
• zxJDBC
• DB-API 2.0 Written in Java using JDBC API
so can utilize JDBC drivers
• Support for connection pools and JNDI
lookup
• Included with standard Jython
installation http://www.jython.org/
• jyjdbc
• DB-API 2.0 compliant
• Written in Python/Jython so can utilize
JDBC drivers
• Decimal data type support
• https://bitbucket.org/clach04/jyjdbc/
22. IronPython + SQL
• adodbapi
• IronPython 2+
• Also works with CPython 2.3+ with
pywin32
• http://adodbapi.sourceforge.net/
23. Gerald, the half a schema
import gerald
s1 = gerald.PostgresSchema(’public',
'postgres://ip2country_rw:secret@localhost/ip2country')
s2 = gerald.PostgresSchema(’public',
'postgres://ip2country_rw:secret@localhost/ip2countryv4')
print s1.schema['ip2country'].compare(s2.schema['ip2country'])
DIFF: Definition of assigned is different
DIFF: Column countryname not in ip2country
DIFF: Definition of registry is different
DIFF: Column countrycode3 not in ip2country
DIFF: Definition of countrycode2 is different
• Database schema toolkit
• via DB-API currently supports
• PostgreSQL
• MySQL
• Oracle
• http://halfcooked.com/code/gerald/
24. SQLPython
$ sqlpython --postgresql ip2country ip2country_rw
Password:
0:ip2country_rw@ip2country> select * from ip2country where countrycode2='SG';
...
1728830464.0 1728830719.0 apnic 2011-11-02 SG SGP Singapore
551 rows selected.
0:ip2country_rw@ip2country> select * from ip2country where countrycode2='SG'j
[...
{"ipfrom": 1728830464.0, "ipto": 1728830719.0, "registry": "apnic”,"assigned":
"2011-11-02", "countrycode2": "SG", "countrycode3": "SGP", "countryname":
"Singapore"}]
• A command-line interface to relational
databases
• via DB-API currently supports
• PostgreSQL
• MySQL
• Oracle
• http://packages.python.org/sqlpython/
25. SQLPython, batteries included
0:ip2country_rw@ip2country> select * from ip2country where countrycode2 =’MY’;
...
1728830464.0 1728830719.0 apnic 2011-11-02 MY MYS Malaysia
551 rows selected.
0:ip2country_rw@ip2country> py
Python 2.6.6 (r266:84292, May 20 2011, 16:42:25)
[GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2
py <command>: Executes a Python command.
py: Enters interactive Python mode.
End with `Ctrl-D` (Unix) / `Ctrl-Z` (Windows), `quit()`, 'exit()`.
Past SELECT results are exposed as list `r`;
most recent resultset is `r[-1]`.
SQL bind, substitution variables are exposed as `binds`, `substs`.
Run python code from external files with ``run("filename.py")``
>>> r[-1][-1]
(1728830464.0, 1728830719.0, 'apnic', datetime.date(2011, 11, 2), ’MY', ’MYS',
’Malaysia')
>>> import socket, struct
>>> def num_to_dotted_quad(n):
... return socket.inet_ntoa(struct.pack('!L',n))
...
>>> num_to_dotted_quad(int(r[-1][-1].ipfrom))
'103.11.220.0'
26. SpringPython – Database Templates
# Find ipv4 address ranges assigned to Malaysia
# using SpringPython DatabaseTemplate & DictionaryRowMapper
from springpython.database.core import *
from springpython.database.factory import *
conn_factory = PgdbConnectionFactory(
user="ip2country_rw", password="secret",
host="localhost", database="ip2countrydb")
dt = DatabaseTemplate(conn_factory)
results = dt.query(
"SELECT * FROM ip2country WHERE countrycode2=%s",
(”MY",), DictionaryRowMapper())
for row in results:
print("%s - %s" % (num_to_dotted_quad(int(row['ipfrom'])),
num_to_dotted_quad(int(row['ipto']))))
27. SQLAlchemy
http://www.sqlalchemy.org/
First release in 2005
Now at version 1.0.8
What is it
• Provides helpers, tools & components to
assist with database access
• Provides a consisdent and full featured
façade over the Python DBAPI
• Provides an optional object relational
mapper(ORM)
• Foundation for many Python third party
libraries & tools
• It doesn’t hide the database, you need
understand SQL
29. SQLAlchemy Core – The Engine
from sqlalchemy import create_engine
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/i
p2countrydb')
engine.execute("""
create table registry (
id serial primary key,
name text
)
""")
engine.execute("""
insert into registry(name) values('apnic')
""")
engine.execute("""
insert into registry(name) values('aprn')
""”)
engine.execute("""
insert into registry(name) values('lacnic')
""")
30. SQLAlchemy Core – SQL Expression Language
from sqlalchemy import create_engine, Table, Column, Integer, String,
MetaData
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2countrydb',
echo=True)
metadata = MetaData()
registry = Table('registry', metadata,
Column('id', Integer,
autoincrement=True,
primary_key=True),
Column('name', String(10)))
metadata.create_all(engine) # create table if it doesn't exist
# auto construct insert statement with binding parameters
ins = registry.insert().values(name='dummy’)
conn = engine.connect() # get database connection
# insert multiple rows with explicit commit
conn.execute(ins, [{'name': 'apnic'},
{'name': 'aprn'}, {'name': 'lacnic'}])
31. SQLAlchemy Core – SQL Expression Language
from sqlalchemy import create_engine, Table, Column, Integer, String,
MetaData
from sqlalchemy.sql import select
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2countrydb',
echo=True)
metadata = MetaData()
registry = Table('registry', metadata,
Column('id', Integer, autoincrement=True,
primary_key=True,
Column('name', String(10)))
# auto create select statement
s = select([registry])
conn = engine.connect()
result = conn.execute(s)
for row in result:
print(row)
32. SQLAlchemy Core – SQL Expression Language
from sqlalchemy import create_engine, Table, Column, Integer, String,
MetaData
from sqlalchemy.sql import select
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2countrydb',
echo=True)
metadata = MetaData()
registry = Table('registry', metadata,
Column('id', Integer, autoincrement=True,
primary_key=True,
Column('name', String(10)))
# auto create select statement
s = select([registry])
conn = engine.connect()
result = conn.execute(s)
for row in result:
print(row)
33. SQLAlchemy ORM
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine, Table, Column, Integer, String
Base = declarative_base()
class Registry(Base):
__tablename__ = 'registry'
id = Column(Integer, autoincrement=True, primary_key=True)
name = Column(String(10))
def __repr__(self):
return "<Registry(%r, %r)>" % (
self.id, self.name
)
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2countrydb',
echo=True)
Base.metadata.create_all(engine)
from sqlalchemy.orm import Session
session = Session(bind=engine)
apnic = session.query(Registry).filter_by(name='apnic').first()
print(apnic)
34. SQLAlchemy ORM
. . .
Base = declarative_base()
class Registry(Base):
__tablename__ = 'registry'
id = Column(Integer, autoincrement=True, primary_key=True)
name = Column(String(10))
def __repr__(self):
return "<Registry(%r, %r)>" % (
self.id, self.name
)
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2countrydb',
echo=True)
Base.metadata.create_all(engine)
from sqlalchemy.orm import Session
session = Session(bind=engine)
mynic = Registry(name='mynic')
session.add(mynic)
36. Andy Todd’s OSDC paper
http://halfcooked.com/presentations/osdc2006/p
ython_databases.html
Source of csv data used in examples from
WebNet77 licensed under GPLv3
http://software77.net/geo-ip/
Attributions
37. Mark Rees
mark at censof dot com
+Mark Rees
@hexdump42
hex-dump.blogspot.com
Contact Details
Hinweis der Redaktion
If you were at PyCon APAC 2012, the first part of this talk will a case of déjà vu if you attended my talk there.
For some Python programmers, their only exposure to accessing relational data is via a object relational mapper (ORM). As powerful is the concept of mapping objects to data, sometimes it is much simpler to manipulate your relational data using SQL. This talk will be about using the DB-API, Python’s standard mechanism for accessing relational databases.
Or maybe you prefer sqlalchemy to abstract away the database. Both the Django ORM and SQLAlchemy need a lower level API to access databases. So the first part of this talk will be about using the DB-API, Python’s standard mechanism for accessing relational databases.
SQL (Structured Query Language) is a DSL and we can achieve the same results as the previous two slides. This what DBA’s program in.
This diagram no longer seems to exist on Travis’s site
Always use parameter binding. Why?
* you normally get better performance from some database engines due to to SQL query caching
* reduce the chance of SQL injection
Always use parameter binding. Why?
* you normally get better performance from some database engines due to to SQL query caching
* reduce the chance of SQL injection
Gerald is a general purpose database schema toolkit written in Python. It can be used for cataloguing, managing and deploying database schemas. It is designed to allow you to easily identify the differences between databases.
SQLPython is a command-line interface to relational databases written in Python. It was created as an alternative to Oracle’s SQL\*Plus, and can likewise be used instead of postgres’ psql or mysql’s mysql text clients. In addition, it offers several extra features inspired by other command-line clients: Neatened output, smart prompt, tab completion, history, scripting, output to file, paste buffer & os command, unix like commands – ls cat grep, data dictionary exploration. Another feature is special output formats. By replacing the ; that terminates a SELECT statement with a backslash-character sequence, you can get output in a number of useful formats like xml, json, csv etc
One of the most powerful features is the py command. The py command allows the user to execute Python commands, either one-at-a-time (with py {command}) or in an interactive environment (beginning with a bare py statement, and continuing until Ctrl-D, quit(), or exit() is entered). A history of result sets from each query is exposed to the python session as the list r; the most recent result set is r[-1]. Each row can be references as a tuple, or as an object with an attribute for each column.
Spring Python takes the concepts of the Spring Framework and Spring Security, and brings them to the world of Python. It isn't a simple line-by-line port of the code. Instead, it takes some powerful ideas that were discovered in the realm of Java, and pragmatically applies them in the world of Python.
One of these paradigms is a Portable Service Abstraction called DatabaseTemplate.
* It is portable because it uses Python's standardized API, not tying us to any database vendor. Instead, in our example, we injected in an instance of Sqlite3ConnectionFactory
* It provides the useful service of easily accessing information stored in a relational database, but letting us focus on the query, not the plumbing code
* It offers a nice abstraction over Python's low level database API with reduced code noise. This allows us to avoid the cost and risk of writing code to manage cursors and exception handling
DatabaseTemplate handles exceptions by catching and holding them, then properly closing the cursor. It then raises it wrapped inside a Spring Python DataAccessException. This way, database resources are properly disposed of without losing the exception stack trace.
The Database Template can be used in isolation from the SpringPython framework.
Core
Engine – a registry which provides connectivity to a particular database server
Dialect – interprets generic SQL and database commands to match a specific DBAPI & database backend
Connection Pool – a collection of database connections in memory for fast re-use
SQL Expression Language – Allows SQL statements to be written using Python expressions
Schema/Types – Uses objects to represent tables, columns and data types
ORM
Allows construction of Python objects which can be mapped to relational database tables
Transparently persists objects into their corresponding database tables using the unit of work pattern
Provides a query system which loads objects and attributes using SQL generated from mappings
Built of top of the Core, uses core to generate SQL and talk to DB
Not much different from using the DBAPI directly
But using the SQL Expression Language it is more pythonic
You can do the same for selects
You can do the same for selects
You use a session to use the unit of work pattern
The session caches the add and doesn’t do the insert until the session is committed.