SQLAthanor ¶

Serialization/De-serialization Support for the SQLAlchemy Declarative ORM

Branch	Unit Tests
latest
v.0.2.0
v.0.1.1
v.0.1.0
develop

Quickstart: Patterns and Best Practices¶

Installation
Meta Configuration Pattern
Declarative Configuration Pattern
Serializing Model Instances
Updating a Model Instance
Creating a New Model Instance
Error Handling
- Errors During Serialization
- Errors During De-serialization
Password De-serialization
Using SQLAthanor with SQLAlchemy Reflection
Using SQLAthanor with Automap
Using SQLAthanor with Flask-SQLAlchemy

Installation ¶

To install SQLAthanor, just execute:

$ pip install sqlathanor

Meta Configuration Pattern ¶

See also

Configuring Serialization/De-serialization > Meta Configuration

from sqlathanor import declarative_base, Column, relationship, AttributeConfiguration

from sqlalchemy import Integer, String
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.ext.associationproxy import association_proxy

BaseModel = declarative_base()

class User(BaseModel):
  __tablename__ = 'users'

  __serialization__ = [AttributeConfiguration(name = 'id',
                                              supports_csv = True,
                                              csv_sequence = 1,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None),
                       AttributeConfiguration(name = 'addresses',
                                              supports_json = True,
                                              supports_yaml = (True, True),
                                              supports_dict = (True, False),
                                              on_serialize = None,
                                              on_deserialize = None),
                       AttributeConfiguration(name = 'hybrid',
                                              supports_csv = True,
                                              csv_sequence = 2,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None)]
                       AttributeConfiguration(name = 'keywords',
                                              supports_csv = False,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None)]
                       AttributeConfiguration(name = 'python_property',
                                              supports_csv = (False, True),
                                              csv_sequence = 3,
                                              supports_json = (False, True),
                                              supports_yaml = (False, True),
                                              supports_dict = (False, True),
                                              on_serialize = None,
                                              on_deserialize = None)]

  id = Column('id',
              Integer,
              primary_key = True)

  addresses = relationship('Address',
                           backref = 'user')

  _hybrid = 1

  @hybrid_property
  def hybrid(self):
      return self._hybrid

  @hybrid.setter
  def hybrid(self, value):
      self._hybrid = value

  @hybrid.expression
  def hybrid(cls):
    return False

  keywords = association_proxy('keywords', 'keyword')

  @property
  def python_property(self):
    return self._hybrid * 2

Declarative Configuration Pattern ¶

See also

Configuring Serialization/De-serialization > Declarative Configuration

from sqlathanor import declarative_base, Column, relationship

from sqlalchemy import Integer, String

BaseModel = declarative_base()

class User(BaseModel):
  __tablename__ = 'users'

  id = Column('id',
              Integer,
              primary_key = True,
              supports_csv = True,
              csv_sequence = 1,
              supports_json = True,
              supports_yaml = True,
              supports_dict = True,
              on_serialize = None,
              on_deserialize = None)

  addresses = relationship('Address',
                           backref = 'user',
                           supports_json = True,
                           supports_yaml = (True, True),
                           supports_dict = (True, False),
                           on_serialize = None,
                           on_deserialize = None)

Serializing Model Instances ¶

See also

# For a SQLAlchemy Model Class named "User" with an instance named "user":

as_csv = user.to_csv()     # CSV
as_json = user.to_json()   # JSON
as_yaml = user.to_yaml()   # YAML
as_dict = user.to_dict()   # dict

Updating a Model Instance ¶

See also

# For a SQLAlchemy Model Class named "User" with an instance named "user"
# and serialized objects "as_csv" (string), "as_json" (string),
# "as_yaml" (string), and "as_dict" (dict):

user.update_from_csv(as_csv)   # CSV
user.update_from_json(as_json) # JSON
user.update_from_yaml(as_yaml) # YAML
user.update_from_dict(as_dict) # dict

Creating a New Model Instance ¶

See also

# For a SQLAlchemy Model Class named "User" and serialized objects "as_csv"
# (string), "as_json" (string), "as_yaml" (string), and "as_dict" (dict):

user = User.new_from_csv(as_csv)   # CSV
user = User.new_from_json(as_json) # JSON
user = User.new_from_yaml(as_yaml) # YAML
user = User.new_from_dict(as_dict) # dict

Error Handling ¶

Errors During Serialization ¶

See also

from sqlathanor.errors import SerializableAttributeError, \
  UnsupportedSerializationError, MaximumNestingExceededError

# For a SQLAlchemy Model Class named "User" and a model instance named "user".

try:
  as_csv = user.to_csv()
  as_json = user.to_json()
  as_yaml = user.to_yaml()
  as_dict = user.to_dict()
except SerializableAttributeError as error:
  # Handle the situation where "User" model class does not have any attributes
  # serializable to JSON.
  pass
except UnsupportedSerializationError as error:
  # Handle the situation where one of the "User" model attributes is of a data
  # type that does not support serialization.
  pass
except MaximumNestingExceededError as error:
  # Handle a situation where "user.to_json()" received max_nesting less than
  # current_nesting.
  #
  # This situation is typically an error on the programmer's part, since
  # SQLAthanor by default avoids this kind of situation.
  #
  # Best practice is simply to let this exception bubble up.
  raise error

Errors During De-serialization ¶

See also

from sqlathanor.errors import DeserializableAttributeError, \
  CSVStructureError, DeserializationError, ValueDeserializationError, \
  ExtraKeysError, UnsupportedDeserializationError

# For a SQLAlchemy Model Class named "User" and a model instance named "user",
# with serialized data in "as_csv", "as_json", "as_yaml", and "as_dict" respectively.

try:
  user.update_from_csv(as_csv)
  user.update_from_json(as_json)
  user.update_from_yaml(as_yaml)
  user.update_from_dict(as_dict)

  new_user = User.new_from_csv(as_csv)
  new_user = User.new_from_json(as_json)
  new_user = User.new_from_yaml(as_yaml)
  new_user = User.new_from_dict(as_dict)
except DeserializableAttributeError as error:
  # Handle the situation where "User" model class does not have any attributes
  # de-serializable from the given format (CSV, JSON, YAML, or dict).
  pass
except DeserializationError as error:
  # Handle the situation where the serialized object ("as_csv", "as_json",
  # "as_yaml", "as_dict") cannot be parsed, for example because it is not
  # valid JSON, YAML, or dict.
  pass
except CSVStructureError as error:
  # Handle the situation where the structure of "as_csv" does not match the
  # expectation configured for the "User" model class.
  raise error
except ExtraKeysError as error:
  # Handle the situation where the serialized object ("as_json",
  # "as_yaml", "as_dict") may have unexpected keys/attributes and
  # the error_on_extra_keys argument is False.
  #
  # Applies to: *_from_json(), *_from_yaml(), and *_from_dict() methods
  pass
except ValueDeserializationError as error:
  # Handle the situation where an input value in the serialized object
  # raises an exception in the deserialization post-processing function.
  pass
except UnsupportedDeserializationError as error:
  # Handle the situation where the de-serialization process attempts to
  # assign a value to an attribute that does not support de-serialization.
  pass

Password De-serialization ¶

See also

from sqlathanor import declarative_base, Column, AttributeConfiguration

from sqlalchemy import Integer, String

def my_encryption_function(value):
  """Function that accepts an inbound password ``value`` and returns its
  encrypted value."""

  # ENCRYPTION LOGIC GOES HERE

  return encrypted_value

BaseModel = declarative_base()

class User(BaseModel):
  __tablename__ = 'users'

  __serialization__ = [AttributeConfiguration(name = 'id',
                                              supports_csv = True,
                                              csv_sequence = 1,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None),
                       AttributeConfiguration(name = 'password',
                                              supports_csv = (True, False),
                                              supports_json = (True, False),
                                              supports_yaml = (True, False),
                                              supports_dict = (True, False),
                                              on_serialize = None,
                                              on_deserialize = my_encryption_function)]

  id = Column('id',
              Integer,
              primary_key = True)

  password = Column('password', String(255))

from sqlathanor import declarative_base, Column

from sqlalchemy import Integer, String

def my_encryption_function(value):
  """Function that accepts an inbound password ``value`` and returns its
  encrypted value."""

  # ENCRYPTION LOGIC GOES HERE

  return encrypted_value

BaseModel = declarative_base()

class User(BaseModel):
  __tablename__ = 'users'

  id = Column('id',
              Integer,
              primary_key = True,
              supports_csv = True,
              csv_sequence = 1,
              supports_json = True,
              supports_yaml = True,
              supports_dict = True,
              on_serialize = None,
              on_deserialize = None)

  password = Column('password',
                    String(255),
                    supports_csv = (True, False),
                    csv_sequence = 2,
                    supports_json = (True, False),
                    supports_yaml = (True, False),
                    supports_dict = (True, False),
                    on_serialize = None,
                    on_deserialize = my_encryption_function)

Using SQLAthanor with SQLAlchemy Reflection ¶

See also

Using Declarative Reflection with SQLAthanor
SQLAlchemy: Reflecting Database Objects
SQLAlchemy: Using Reflection with Declarative

from sqlathanor import declarative_base, Column, AttributeConfiguration

from sqlalchemy import create_engine, Table

BaseModel = declarative_base()

engine = create_engine('... ENGINE CONFIGURATION GOES HERE ...')
# NOTE: Because reflection relies on a specific SQLAlchemy Engine existing, presumably
# you would know how to configure / instantiate your database engine using SQLAlchemy.
# This is just here for the sake of completeness.

class ReflectedUser(BaseModel):
    __table__ = Table('users',
                      BaseModel.metadata,
                      autoload = True,
                      autoload_with = engine)

    __serialization__ = [AttributeConfiguration(name = 'id',
                                                supports_csv = True,
                                                csv_sequence = 1,
                                                supports_json = True,
                                                supports_yaml = True,
                                                supports_dict = True,
                                                on_serialize = None,
                                                on_deserialize = None),
                         AttributeConfiguration(name = 'password',
                                                supports_csv = (True, False),
                                                supports_json = (True, False),
                                                supports_yaml = (True, False),
                                                supports_dict = (True, False),
                                                on_serialize = None,
                                                on_deserialize = None)]

    # ADDITIONAL RELATIONSHIPS, HYBRID PROPERTIES, OR ASSOCIATION PROXIES
    # GO HERE

from sqlathanor import declarative_base, Column, AttributeConfiguration

from sqlalchemy import create_engine, Table, Integer, String

BaseModel = declarative_base()

engine = create_engine('... ENGINE CONFIGURATION GOES HERE ...')
# NOTE: Because reflection relies on a specific SQLAlchemy Engine existing, presumably
# you would know how to configure / instantiate your database engine using SQLAlchemy.
# This is just here for the sake of completeness.

UserTable = Table('users',
                  BaseModel.metadata,
                  Column('id',
                         Integer,
                         primary_key = True,
                         supports_csv = True,
                         csv_sequence = 1,
                         supports_json = True,
                         supports_yaml = True,
                         supports_dict = True,
                         on_serialize = None,
                         on_deserialize = None),
                  Column('password',
                         String(255),
                         supports_csv = (True, False),
                         csv_sequence = 2,
                         supports_json = (True, False),
                         supports_yaml = (True, False),
                         supports_dict = (True, False),
                         on_serialize = None,
                         on_deserialize = None))

class ReflectedUser(BaseModel):
    __table__ = Table('users',
                      BaseModel.metadata,
                      autoload = True,
                      autoload_with = engine)

    # ADDITIONAL RELATIONSHIPS, HYBRID PROPERTIES, OR ASSOCIATION PROXIES
    # GO HERE

Tip

In practice, this pattern eliminates the time-saving benefits of using reflection in the first place. Instead, I would recommend adopting the meta configuration pattern with reflection instead.

from sqlathanor import declarative_base, Column, AttributeConfiguration

from sqlalchemy import create_engine, Table, Integer, String

BaseModel = declarative_base()

engine = create_engine('... ENGINE CONFIGURATION GOES HERE ...')
# NOTE: Because reflection relies on a specific SQLAlchemy Engine existing, presumably
# you would know how to configure / instantiate your database engine using SQLAlchemy.
# This is just here for the sake of completeness.

class ReflectedUser(BaseModel):
    __table__ = Table('users',
                      BaseModel.metadata,
                      autoload = True,
                      autoload_with = engine)

    id = Column('id',
                Integer,
                primary_key = True,
                supports_csv = True,
                csv_sequence = 1,
                supports_json = True,
                supports_yaml = True,
                supports_dict = True,
                on_serialize = None,
                on_deserialize = None)

    password = Column('password',
                      String(255),
                      supports_csv = (True, False),
                      csv_sequence = 2,
                      supports_json = (True, False),
                      supports_yaml = (True, False),
                      supports_dict = (True, False),
                      on_serialize = None,
                      on_deserialize = None)

    # ADDITIONAL RELATIONSHIPS, HYBRID PROPERTIES, OR ASSOCIATION PROXIES
    # GO HERE

Using SQLAthanor with Automap ¶

New in version 0.2.0.

See also

Using Automap with SQLAthanor
SQLAlchemy: Automap Extension

Error

If you try to use automap_base() with SQLAlchemy v.0.9.0, you will get a SQLAlchemySupportError.

from sqlathanor.automap import automap_base
from sqlalchemy import create_engine

# Create your Automap Base
Base = automap_base()

engine = create_engine('... DATABASE CONNECTION GOES HERE ...')

# Prepare your automap base. This reads your database and creates your models.
Base.prepare(engine, reflect = True)

# And here you can create a "User" model class and an "Address" model class.
User = Base.classes.users
Address = Base.classes.addresses


User.set_attribute_serialization_config('email_address',
                                        supports_csv = True,
                                        supports_json = True,
                                        supports_yaml = True,
                                        supports_dict = True)
User.set_attribute_serialization_config('password',
                                        supports_csv = (True, False),
                                        supports_json = (True, False),
                                        supports_yaml = (True, False),
                                        supports_dict = (True, False),
                                        on_deserialize = my_encryption_function)

from sqlathanor.automap import automap_base
from sqlalchemy import create_engine

# Create your Automap Base
Base = automap_base()

engine = create_engine('... DATABASE CONNECTION GOES HERE ...')

# Prepare your automap base. This reads your database and creates your models.
Base.prepare(engine, reflect = True)

# And here you can create a "User" model class and an "Address" model class.
User = Base.classes.users
Address = Base.classes.addresses

User.__serialization__ = [
    {
        'name': 'email_address',
        'supports_csv': True,
        'supports_json': True,
        'supports_yaml': True,
        'supports_dict': True
    },
    {
        'name': 'password',
        'supports_csv': (True, False),
        'supports_json': (True, False),
        'supports_yaml': (True, False),
        'supports_dict': (True, False),
        'on_deserialize': my_encryption_function
    }
]

Using SQLAthanor with Flask-SQLAlchemy ¶

See also

Import SQLAthanor > Using Flask-SQLAlchemy
Flask-SQLAlchemy Documentation

from sqlathanor import FlaskBaseModel, initialize_sqlathanor
from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy(model_class = FlaskBaseModel)
db = initialize_sqlathanor(db)

Using SQLAthanor¶

Introduction
SQLAthanor Features
Overview
- How SQLAthanor Works
1. Installing SQLAthanor
- Dependencies
2. Import SQLAthanor
3. Define Your Models
4. Configure Serialization/De-serialization
5. Configuring Pre-processing and Post-processing
- Serialization Pre-processing
- De-serialization Post-processing
6. Serializing a Model Instance
7. Deserializing Data
- Creating New Instances
- Updating Instances
Using Declarative Reflection with SQLAthanor
Using Automap with SQLAthanor

Introduction ¶

What is Serialization?¶

“Serialization” is a common short-hand for two important concepts: serialization and de-serialization.

To over-simplify, serialization is the process of taking an object in one programming language (say, Python) that can be understood by one program (say, yours) and converting it into a format that can be understood by a different program, often written in a different programming language (say, JavaScript).

De-serialization is the exact same process - but in reverse. It takes data in another format, often from another program, and converts it into an object that your program can understand.

So why is it important? Well, because modern programming is all about communication. Which in this context means communication between different programs, APIs, microservices, etc. And these various programs and microservices may be written in vastly different programming languages, have (often different) approaches to security, etc. Which means they need to be able to talk to each other appropriately.

Which is where serialization and de-serialization come in, since they’re the process that makes that communication possible.

In a very typical example, you can imagine a modern web application. The back-end might be written in Python, maybe using a framework like Flask or Django. The back-end exposes a variety of RESTful APIs that handle the business logic of your web app. But you’ve got an entirely separate front-end, probably written in JavaScript using React/Redux, AngularJS, or something similar.

In most web applications, at some point your back-end will need to retrieve data from a database (which an ORM like SQLAlchemy is great at), and will want to hand it off to your front-end. A typical example might be if you want to list users, or in a social media-style app, list a user’s friends. So once your Python back-end has gotten a list of users, how does it communicate that list to your JavaScript front-end? Most likely by exchanging JSON objects.

Which means that your Python back-end needs to take the list of users it retrieved, convert their data into JSON format, and transmit it to the front-end. That’s serialization.

But now let’s say a user in your front-end changes their email address. The front-end will need to let the back-end know, and your back-end will need to update the relevant database record with the latest change. So how does the front-end communicate that change to the back-end? Again, by sending a JSON object to the back-end. But your back-end needs to parse that data, validate it, and then reflect the change in the underlying database. The process of parsing that data? That’s de-serialization.

Why SQLAthanor?¶

So if serialziation and de-serialization are so important, how does this relate to SQLAthanor? Well, serialization and de-serialization can be complicated:

Different programs may need to serialize and de-serialize into and from multiple formats.
Some data (like passwords) should only be de-serialized, but for security reasons should never be serialized.
Serialization and de-serialization may need various pre-processing steps to validate the data or coerce it to/from different data types…and that validation/coercion logic may be different depending on the data format.
The (fantastic) SQLAlchemy ORM handles database read/write operations amazingly, but does not include any serialization/de-serialization support.

This leads to a labor-intensive process of writing custom serialization/de-serialization logic for multiple (different) models and repeating that process across multiple applications. Better, we think, to package that functionality into a library.

Which is what SQLAthanor is.

It is designed to extend the functionality of the SQLAlchemy ORM with support for serialization and de-serialization into/from:

JSON
CSV
YAML
Python dict

Which should hopefully save some effort when building applications that need to talk to the outside world (and really, don’t all apps do these days?).

SQLAthanor vs. Alternatives ¶

Since serialization and de-serialization are common problems, there are a variety of alternative ways to serialize and de-serialize your SQLAlchemy models. Obviously, I’m biased in favor of SQLAthanor. But it might be helpful to compare SQLAthanor to some commonly-used alternatives:

Adding your own custom serialization/de-serialization logic to your SQLAlchemy declarative models is a very viable strategy. It’s what I did for years, until I got tired of repeating the same patterns over and over again, and decided to build SQLAthanor instead.

But of course, implementing custom serialization/de-serialization logic takes a bit of effort.

Tip

When to use it?

In practice, I find that rolling my own solution is great when it’s a simple model with very limited business logic. It’s a “quick-and-dirty” solution, where I’m trading rapid implementation (yay!) for less flexibility/functionality (boo!).

Considering how easy SQLAthanor is to configure / apply, however, I find that I never really roll my own serialization/de-serialization approach when working SQLAlchemy models any more.

The Marshmallow library and its Marshmallow-SQLAlchemy extension are both fantastic. However, they have one major architectural difference to SQLAthanor and several more minor differences:

The biggest difference is that by design, they force you to maintain two representations of your data model. One is your SQLAlchemy model class, while the other is your Marshmallow schema (which determines how your model is serialized/de-serialized). Marshmallow-SQLAlchemy specifically tries to simplify this by generating a schema based on your model class, but you still need to configure, manage, and maintain both representations - which as your project gets more complex, becomes non-trivial.

SQLAthanor by contrast lets you configure serialization/deserialization in your SQLAlchemy model class definition. You’re only maintaining one data model representation in your Python code, which is a massive time/effort/risk-reduction.

Other notable differences relate to the API/syntax used to support non-JSON formats. I think Marshmallow uses a non-obvious approach, while with SQLAthanor the APIs are clean and simple. Of course, on this point, YMMV.

Tip

When to use it?

Marshmallow has one advantage over SQLAthanor: It can serialize/de-serialize any Python object, whether it is a SQLAlchemy model class or not. SQLAthanor only works with SQLAlchemy.

As a result, it may be worth using Marshmallow instead of SQLAthanor if you expect to be serializing / de-serializing a lot of non-SQLAlchemy objects.

The Colander library and the ColanderAlchemy extension are both great, but they have a similar major architectural difference to SQLAthanor as Marshmallow/Marshmallow-SQLAlchemy:

By design, they force you to maintain two representations of your data model. One is your SQLAlchemy model class, while the other is your Colander schema (which determines how your model is serialized/de-serialized). ColanderAlchemy tries to simplify this by generating a schema based on your model class, but you still need to configure, manage, and maintain both representations - which as your project gets more complex, becomes non-trivial.

SQLAthanor by contrast lets you configure serialization/deserialization in your SQLAlchemy model class definition. You’re only maintaining one data model representation in your Python code, which is a massive time/effort/risk-reduction.

A second major difference is that, again by design, Colander is designed to serialize/de-serialize Python objects to a set of Python primitives. Since neither JSON, CSV, or YAML are Python primitives, you’ll still need to serialize/de-serialize Colander’s input/output to/from its final “transmissable” form. Once you’ve got a Python primitive, this isn’t difficult - but it is an extra step.

Tip

When to use it?

Colander has one advantage over SQLAthanor: It can serialize/de-serialize any Python object, whether it is a SQLAlchemy model class or not. SQLAthanor only works with SQLAlchemy.

As a result, it may be worth using Colander instead of SQLAthanor if you expect to be serializing / de-serializing a lot of non-SQLAlchemy objects.

pandas is one of my favorite analytical libraries. It has a number of great methods that adopt a simple syntax, like read_csv() or to_csv() which de-serialize / serialize data to various formats (including SQL, JSON, CSV, etc.).

So at first blush, one might think: Why not just use pandas to handle serialization/de-serialization?

Well, pandas isn’t really a serialization alternative to SQLAthanor. More properly, it is an ORM alternative to SQLAlchemy itself.

I could write (and have written) a lot on the subject, but the key difference is that pandas is a “lightweight” ORM that focuses on providing a Pythonic interface to work with the output of single SQL queries. It does not support complex relationships between tables, or support the abstracted definition of business logic that applies to an object representation of a “concept” stored in your database.

SQLAlchemy is specifically designed to do those things.

So you can think of pandas as being a less-abstract, “closer to bare metal” ORM - which is what you want if you want very efficient computations, on relatively “flat” (non-nested/minimally relational) data. Modification or manipulation of the data can be done by mutating your pandas DataFrame without too much maintenance burden because those mutations/modifications probably don’t rely too much on complex abstract business logic.

SQLAthanor piggybacks on SQLAlchemy’s business logic-focused ORM capabilities. It is designed to allow you to configure expected behavior once and then re-use that capability across all instances (records) of your data. And it’s designed to play well with all of the other complex abstractions that SQLAlchemy supports, like relationships, hybrid properties, reflection, or association proxies.

pandas serialization/de-serialization capabilities can only be configured “at use-time” (in the method call), which leads to a higher maintenance burden. SQLAthanor’s serialization/de-serialization capabilities are specifically designed to be configurable when defining your data model.

Tip

When to use it?

The decision of whether to use pandas or SQLAlchemy is a complex one, but in my experience a good rule of thumb is to ask yourself whether you’re going to need to apply complex business logic to your data.

The more complex the business logic is, the more likely SQLAlchemy will be a better solution. And if you are using SQLAlchemy, then SQLAthanor provides great and easy-to-use serialization/de-serialization capabilities.

SQLAthanor Features ¶

Configure serialization and de-serialization support when defining your SQLAlchemy models.
Automatically include serialization methods in your SQLAlchemy model instances.
Automatically include de-serialization “creator” methods in your SQLAlchemy models.
Automatically include de-serialization “updater” methods to your SQLAlchemy model instances.
Support serialization and de-serialization across the most-common data exchange formats: JSON, CSV, YAML, and Python dict.
Support pre-processing before serialization/de-serialization for data validation or coercion.
Support serialization and de-serialization for complex models that may include: relationships, hybrid properties, association proxies, or standard Python @property.
Maintain all of the existing APIs, methods, functions, and functionality of SQLAlchemy Core.
Maintain all of the existing APIs, methods, functions, and functionality of SQLAlchemy ORM.
Maintain all of the existing APIs, methods, functions, and functionality of SQLAlchemy Declarative ORM.

Overview ¶

SQLAthanor is designed to extend the fantastic SQLAlchemy library, to provide it with seamless serialization and de-serialization support. What do we mean by seamless? Well, in an ideal world we want serialization and de-serialization to work like this:

# To create serialized output from a model instance, just use:
as_json = model_instance.to_json()
as_csv = model_instance.to_csv()
as_yaml = model_instance.to_yaml()
as_dict = model_instance.to_dict()

# To create a new model instance from serialized data, just use:
new_as_instance = ModelClass.new_from_json(as_json)
new_as_instance = ModelClass.new_from_csv(as_csv)
new_as_instance = ModelClass.new_from_yaml(as_yaml)
new_as_instance = ModelClass.new_from_dict(as_dict)

# To update an existing model instance from serialized data, just use:
model_instance.update_from_json(as_json)
model_instance.update_from_csv(as_csv)
model_instance.update_from_yaml(as_yaml)
model_instance.update_from_dict(as_dict)

Even knowing nothing about SQLAlchemy or SQLAthanor, it should be pretty easy to figure out what’s happening in that code, right?

Well, that’s exactly what SQLAthanor does for you. So how? Let’s break that down.

How SQLAthanor Works ¶

SQLAthanor is a drop-in replacement for SQLAlchemy.

What does this mean? It means that it’s designed to seamlessly replace some of your SQLAlchemy import statements. Then, you can continue to define your models just as you would using the SQLAlchemy ORM - but now they’ll support serialization and de-serialization.

In other words, the process of using SQLAthanor is very simple:

Install SQLAthanor. (see here)
Import the components used to define your model classes. (see here)
Define your model classes, just as you would in SQLAlchemy. (see here)
Configure which model attributes to be serialized (output) and de-serialized (input). (see here)
Configure any pre/post-processing for serialization and de-serialization, respectively. (see here)
Serialize your model instances as needed. (see here)
Create new model instances using de-serialization. (see here)
Update existing model instances using de-serialization. (see here)

And that’s it! Once you’ve done the steps above, you can easily serialize data from your models and de-serialize data into your models using simple methods.

Tip

Because SQLAthanor inherits from and extends SQLAlchemy, your existing SQLAlchemy models will work with no change.

By default, serialization and de-serialization are disabled for any model attribute unless they are explicitly enabled.

1. Installing SQLAthanor ¶

To install SQLAthanor, just execute:

$ pip install sqlathanor

Dependencies ¶

2. Import SQLAthanor ¶

Since SQLAthanor is a drop-in replacement, you should import it using the same elements as you would import from SQLAlchemy:

The code below is a pretty standard set of import statements when working with SQLAlchemy and its Declarative ORM.

They’re provided for reference below, but do not make use of SQLAthanor and do not provide any support for serialization or de-serialization:

from sqlalchemy.ext.declarative import declarative_base, as_declarative
from sqlalchemy import Column, Integer, String          # ... and any other data types

# The following are optional, depending on how your data model is designed:
from sqlalchemy.orm import relationship
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.ext.associationproxy import association_proxy

To import SQLAthanor, just replace the relevant SQLAlchemy imports with their SQLAthanor counterparts as below:

from sqlathanor import declarative_base, as_declarative
from sqlathanor import Column
from sqlathanor import relationship             # This import is optional, depending on
                                                # how your data model is designed.

from sqlalchemy import Integer, String          # ... and any other data types

# The following are optional, depending on how your data model is designed:
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.ext.associationproxy import association_proxy

Tip

Because of its many moving parts, SQLAlchemy splits its various pieces into multiple modules and forces you to use many import statements.

The example above maintains this strategy to show how SQLAthanor is a 1:1 drop-in replacement. But obviously, you can import all of the items you need in just one import statement:

from sqlathanor import declarative_base, as_declarative, Column, relationship

SQLAthanor is designed to work with Flask-SQLAlchemy too! However, you need to:

Import the FlaskBaseModel class, and then supply it as the model_class argument when initializing Flask-SQLAlchemy.
Initialize SQLAthanor on your db instance using initialize_sqlathanor.

from sqlathanor import FlaskBaseModel, initialize_sqlathanor
from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy(model_class = FlaskBaseModel)
db = initialize_sqlathanor(db)

And that’s it! Now SQLAthanor serialization functionality will be supported by:

Flask-SQLAlchemy’s db.Model
Flask-SQLAlchemy’s db.relationship()
Flask-SQLAlchemy’s db.Column

See also

For more information about working with Flask-SQLAlchemy, please review their detailed documentation.

As the examples provided above show, importing SQLAthanor is very straightforward, and you can include it in an existing codebase quickly and easily. In fact, your code should work just as before. Only now it will include new functionality to support serialization and de-serialization.

The table below shows how SQLAlchemy classes and functions map to their SQLAthanor replacements:

SQLAlchemy Component	SQLAthanor Analog
`declarative_base()` from sqlalchemy.ext.declarative import declarative_base	`declarative_base()` from sqlathanor import declarative_base
`@as_declarative` from sqlalchemy.ext.declarative import as_declarative	`@as_declarative` from sqlathanor import as_declarative
`Column` from sqlalchemy import Column	`Column` from sqlathanor import Column
`relationship()` from sqlalchemy import relationship	`relationship()` from sqlathanor import relationship
`ext.automap.automap_base()` from sqlalchemy.ext.automap import automap_base	`automap.automap_base()` from sqlathanor.automap import automap_base

3. Define Your Models ¶

Because SQLAthanor is a drop-in replacement for SQLAlchemy and its Declarative ORM, you can define your models the exact same way you would do so normally:

See also

SQLAlchemy supports the use of reflection with the SQLAlchemy Declarative ORM.

This is a process where SQLAlchemy automatically constructs a Declarative model class based on what it reads from the table definition stored in your SQL database or a corresponding Table instance already defined and registered with a MetaData object.

SQLAthanor also supports the same pattern. For details, please see: Using Declarative Reflection with SQLAthanor

See also

Using Declarative Reflection with SQLAthanor
SQLAlchemy: Reflecting Database Objects
SQLAlchemy: Using Reflection with Declarative

New in version 0.2.0.

The Automap Extension is an incredibly useful tool for modeling existing databases with minimal effort. What it does is it reasd your existing database’s metadata and automatically constructs SQLAlchemy Declarative ORM model classes populated with your tables’ columns.

Neat, right? Saves a ton of effort.

Using SQLAthanor you can ensure that your automapped (automatically generated) models support serialization and de-serialization. For more details, please see: Using Automap with SQLAthanor.

See also

Using Automap with SQLAthanor
SQLAlchemy: Automap Extension

4. Configure Serialization/De-serialization ¶

explicit is better than implicit

—PEP 20 - The Zen of Python

By default (for security reasons) serialization and de-serialization are disabled on all of your model attributes. However, SQLAthanor exists to let you explicitly enable serialization and/or de-serialization for specific model attributes.

Here are some important facts to understand for context:

You can enable serialization separately from de-serialization. This allows you to do things like de-serialize a password field (expect it as an input), but never include it in the output that you serialize.

You can configure serialization/de-serialization differently for different formats. SQLAthanor supports CSV, JSON, YAML, and Python dict.

You can serialize or de-serialize any model attribute that is bound to your model class. This includes:

Attributes that correspond to columns in the underlying SQL table.

Attributes that represent a relationship to another model.

Attributes that are defined as hybrid properties.

Attributes that are defined as association proxies.

Instance attributes defined using Python’s built-in @property decorator.

In general, SQLAthanor supports two different mechanisms to configure serialization/de-serialization: Declarative Configuration and Meta Configuration.

Declarative Configuration ¶

The Declarative Configuration approach is modeled after the SQLAlchemy Declarative ORM itself. It allows you to configure a model attribute’s serialization and de-serialization when defining the model attribute.

Here’s a super-simplified example of how it works:

from sqlathanor import declarative_base, Column, relationship

from sqlalchemy import Integer, String

BaseModel = declarative_base()

class User(BaseModel):
  __tablename__ = 'users'

  id = Column('id',
              Integer,
              primary_key = True,
              supports_csv = True,
              csv_sequence = 1,
              supports_json = True,
              supports_yaml = True,
              supports_dict = True,
              on_serialize = None,
              on_deserialize = None)

This example defines a model class called User which corresponds to a SQL database table named users. The User class defines one model attribute named id (which corresponds to a database Column named id). The database column is an integer, and it operates as the primary key for the database table.

So far, this is all exactly like you would normally see in the SQLAlchemy Declarative ORM.

But, there are some additional arguments supplied to Column: supports_csv, csv_sequence, supports_json, supports_yaml, supports_dict, on_serialize, and on_deserialize. As you can probably guess, these arguments are what configure serialization and de-serialization in SQLAthanor when using declarative configuration.

Here’s what these arguments do:

SQLAthanor Configuration Arguments

Parameters:

supports_csv (bool or tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from CSV format.

If True, can be serialized to CSV and de-serialized from CSV. If False, will not be included when serialized to CSV and will be ignored if present in a de-serialized CSV.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to CSV or de-serialized from CSV.
csv_sequence (int or None) –
Indicates the numbered position that the column should be in in a valid CSV-version of the object. Defaults to None.

Note

If not specified, the column will go after any columns that do have a csv_sequence assigned, sorted alphabetically.

If two columns have the same csv_sequence, they will be sorted alphabetically.
supports_json (bool or tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from JSON format.

If True, can be serialized to JSON and de-serialized from JSON. If False, will not be included when serialized to JSON and will be ignored if present in a de-serialized JSON.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to JSON or de-serialized from JSON.
supports_yaml (bool or tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from YAML format.

If True, can be serialized to YAML and de-serialized from YAML. If False, will not be included when serialized to YAML and will be ignored if present in a de-serialized YAML.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to YAML or de-serialized from YAML.
supports_dict (bool or tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from a Python dict.

If True, can be serialized to dict and de-serialized from a dict. If False, will not be included when serialized to dict and will be ignored if present in a de-serialized dict.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to a dict or de-serialized from a dict.
on_deserialize (callable or dict with formats as keys and values as callables) –
A function that will be called when attempting to assign a de-serialized value to the column. This is intended to either coerce the value being assigned to a form that is acceptable by the column, or raise an exception if it cannot be coerced. If None, the data type’s default on_deserialize function will be called instead.
Tip

If you need to execute different on_deserialize functions for different formats, you can also supply a dict:
```
on_deserialize = {
  'csv': csv_on_deserialize_callable,
  'json': json_on_deserialize_callable,
  'yaml': yaml_on_deserialize_callable,
  'dict': dict_on_deserialize_callable
}
```
Defaults to None.
on_serialize (callable or dict with formats as keys and values as callables) –
A function that will be called when attempting to serialize a value from the column. If None, the data type’s default on_serialize function will be called instead.
Tip

If you need to execute different on_serialize functions for different formats, you can also supply a dict:
```
on_serialize = {
  'csv': csv_on_serialize_callable,
  'json': json_on_serialize_callable,
  'yaml': yaml_on_serialize_callable,
  'dict': dict_on_serialize_callable
}
```
Defaults to None.

When using Declarative Configuration, the exact same arguments can be applied when defining a relationship using relationship() as shown in the expanded example below. Let’s look at a somewhat more complicated example:

from sqlathanor import declarative_base, Column, relationship

from sqlalchemy import Integer, String

BaseModel = declarative_base()

class User(BaseModel):
  __tablename__ = 'users'

  id = Column('id',
              Integer,
              primary_key = True,
              supports_csv = True,
              csv_sequence = 1,
              supports_json = True,
              supports_yaml = True,
              supports_dict = True,
              on_serialize = None,
              on_deserialize = None)

  addresses = relationship('Address',
                           backref = 'user',
                           supports_json = True,
                           supports_yaml = (True, True),
                           supports_dict = (True, False),
                           on_serialize = None,
                           on_deserialize = None)

This example is (obviously) very similar to the previous one. But now we have added a relationship defined using the SQLAthanor relationship() function. This operates exactly as the built-in sqlalchemy.relationship() function. But it has the same set of declarative SQLAthanor configuration attributes.

So in this example, we define a relationship to a different model class called Address, and assign that relationship to the model attribute User.addresses. Given the configuration above, the User model will:

support serializing the id attribute to CSV, JSON, YAML, and dict

support de-serializing the id attribute from CSV, JSON, YAML, and dict

support serializing related addresses to JSON and YAML, but will not include the addresses attribute when serializing to CSV or dict

support de-serializing related addresses from JSON, YAML, and dict, but not from CSV.

Meta Configuration ¶

The Meta Configuration approach is a bit more robust than the Declarative approach. That’s because it supports more model attribute types, including hybrid properties, association proxies, and Python @property instance attributes.

The Meta Configuration approach relies on a special model attribute that you define for your model class: __serialization__. This attribute should contain a list of AttributeConfiguration objects, which are used to indicate the explicit serialization/de-serialization configuration for your model attributes.

Here’s how you would configure an example User model using the Meta Configuration approach:

from sqlathanor import declarative_base, Column, relationship, AttributeConfiguration

from sqlalchemy import Integer, String

BaseModel = declarative_base()

class User(BaseModel):
  __tablename__ = 'users'

  __serialization__ = [AttributeConfiguration(name = 'id',
                                              supports_csv = True,
                                              csv_sequence = 1,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None),
                       AttributeConfiguration(name = 'addresses',
                                              supports_json = True,
                                              supports_yaml = (True, True),
                                              supports_dict = (True, False),
                                              on_serialize = None,
                                              on_deserialize = None)]

  id = Column('id',
              Integer,
              primary_key = True)

  addresses = relationship('Address',
                           backref = 'user')

The __serialization__ attribute contains an explicit configuration for both the id and addresses column. Each AttributeConfiguration object supports the same configuration arguments as are used by the declarative approach, with one addition: It needs a name argument that explicitly indicates the name of the model attribute that is being configured.

Note

The __serialization__ attribute accepts both AttributeConfiguration instances, as well as dict representations of those instances.

If you supply dict configurations, SQLAthanor will automatically convert them to AttributeConfiguration instances.

The __serialization__ below is identical to the one above:

__serialization__ = [
    {
        'name': 'id',
        'supports_csv': True,
        'csv_sequence': 1,
        'supports_json': True,
        'supports_yaml': True,
        'supports_dict': True,
        'on_serialize': None,
        'on_deserialize': None
    },
    {
        'name': 'addresses',
        'supports_json': True,
        'supports_yaml': (True, True),
        'supports_dict': (True, False),
        'on_serialize': None,
        'on_deserialize': None
    }
]

See also

AttributeConfiguration
SQLAthanor Configuration Arguments
Declarative Configuration

Unlike the declarative approach, you can use the __serialization__ attribute to configure serialization and de-serialization for more complex types of model attributes, including hybrid properties, association proxies, and Python @property attributes.

Using the meta configuration approach, you configure these more complex attributes in exactly the same way:

from sqlathanor import declarative_base, Column, relationship, AttributeConfiguration

from sqlalchemy import Integer, String
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.ext.associationproxy import association_proxy

BaseModel = declarative_base()

class User(BaseModel):
  __tablename__ = 'users'

  __serialization__ = [AttributeConfiguration(name = 'id',
                                              supports_csv = True,
                                              csv_sequence = 1,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None),
                       AttributeConfiguration(name = 'addresses',
                                              supports_json = True,
                                              supports_yaml = (True, True),
                                              supports_dict = (True, False),
                                              on_serialize = None,
                                              on_deserialize = None),
                       AttributeConfiguration(name = 'hybrid',
                                              supports_csv = True,
                                              csv_sequence = 2,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None)]
                       AttributeConfiguration(name = 'keywords',
                                              supports_csv = False,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None)]
                       AttributeConfiguration(name = 'python_property',
                                              supports_csv = (False, True),
                                              csv_sequence = 3,
                                              supports_json = (False, True),
                                              supports_yaml = (False, True),
                                              supports_dict = (False, True),
                                              on_serialize = None,
                                              on_deserialize = None)]

  id = Column('id',
              Integer,
              primary_key = True)

  addresses = relationship('Address',
                           backref = 'user')

  _hybrid = 1

  @hybrid_property
  def hybrid(self):
      return self._hybrid

  @hybrid.setter
  def hybrid(self, value):
      self._hybrid = value

  @hybrid.expression
  def hybrid(cls):
    return False

  keywords = association_proxy('keywords', 'keyword')

  @property
  def python_property(self):
    return self._hybrid * 2

This more complicated pattern extends the earlier example with a hybrid property nammed hybrid, an association proxy named keywords, and an instance attribute (defined using @property) named python_property.

The __serialization__ configuration shown ensures that:

hybrid can be serialized to and de-serialized from CSV, JSON, YAML, and dict.
keywords can be serialized to and de-serialized from JSON, YAML, and dict, but cannot be serialized to or de-serialized from CSV.
python_property can be serialized to all formats, but cannot be de-serialized (which makes sense, since it is defined without a setter)

Warning

A configuration found in __serialization__ always takes precedence.

This means that if you mix the declarative and meta approaches, the configuration in __serialization__ will be applied.

Tip

For security reasons, if you don’t explicitly configure serialization/de-serialization for a model attribute using the meta or declarative approach, by default that attribute will not be serialized and will be ignored when de-serializing.

Meta Configuration vs Declarative Configuration ¶

How should you choose between the meta configuration and declarative configuration approach?

Well, to some extent it’s a question of personal preference. For example, I always use the meta approach because I believe it is cleaner, easier to maintain, and more extensible. But as you can probably tell if you look at my code, I tend to be a bit pedantic about such things. There are plenty of times when the declarative approach will make sense and “feel” right.

Here’s a handy flowchart to help you figure out which you should use:

Flowchart for Choosing a SQLAthanor Configuration Approach

Why Two Configuration Approaches?¶

The Zen of Python holds that:

There should be one– and preferably only one –obvious way to do it.

And that is a very wise principle. But there are times when it makes sense to diverge from that principle. I made a conscious choice to support two different configuration mechanisms for several practical reasons:

SQLAlchemy has been around for a long time, and is very popular. There are many existing codebases that might benefit from integrating with SQLAthanor. The meta configuration approach lets users make minimal changes to their existing codebases at minimal risk.

SQLAlchemy is often used in quick-and-dirty projects where the additional overhead of defining an explicit meta configuration in the __serialization__ class attribute will interrupt a programmer’s “flow”. The SQLAlchemy Declarative ORM already provides a great API for defining a model with minimal overhead, so piggybacking on that familiar API will enhance the programmer experience.

The internal mechanics of how SQLAlchemy implements hybrid properties and association proxies are very complicated, and can be mutated at various points in a model class or model instance lifecycle. As a result, supporting them in the declarative configuration approach would have been very complicated, with a lot of exposure to potential edge case errors. But the meta configuration approach neatly avoids those risks.

The only way for the declarative configuration approach to support serialization and de-serialization of model attributes defined using Python’s built-in @property decorator would require extending a feature of the standard library…which I consider an anti-pattern that should be done rarely (if ever).

So given those arguments, you might ask: Why not just use the meta configuration approach, and call it a day? It clearly has major advantages. And yes, you’d be right to ask the question. It does have major advantages, and it is the one I use almost-exclusively in my own code.

But!

I’ve spent close to twenty years in the world of data science and analytics, and I know what the coding practices in that community look like and how SQLAlchemy often gets used “in the wild”. And that experience and knowledge tells me that the declarative approach will just “feel more natural” to a large number of data scientists and developers who have a particular workflow, particular coding style, specific coding conventions, and who often don’t need SQLAlchemy’s more complicated features like hybrid properties or association proxies.

And since SQLAthanor can provide benefits to both “application developers” and “data scientists”, I’ve tried to design an interface that will feel “natural” to both communities.

5. Configuring Pre-processing and Post-processing ¶

When serializing and de-serializing objects, it is often necessary to either convert data to a more-compatible format, or to validate inbound data to ensure it meets your expectations.

SQLAthanor supports this using serialization pre-processing and de-serialization post-processing, as configured in the on_serialize and on_deserialize configuration arguments.

Serialization Pre-processing ¶

The on_deserialize configuration argument allows you to assign a serialization function to a particular model attribute.

If assigned, the serialization function will be called before the model attribute’s value gets serialized into its target format. This is particularly useful when you need to convert a value from its native (in Python) type/format into a type/format that is supported by the serialized data type.

A typical example of this might be converting a None value in Python to an empty string '' that can be included in a CSV record, or ensuring a decimal value is appropriately rounded.

The on_serialize argument expects to receive either a callable (a Python function), or a dict where keys correspond to SQLAthanor’s supported formats and values are the callables to use for that format. Thus:

...
on_serialize = my_serialization_function,
...

will call the my_serialization_function() whenever serializing the value, but

...
on_serialize = {
    'csv': my_csv_serialization_function,
    'json': my_json_serialization_function,
    'yaml': my_yaml_serialization_function,
    'dict': my_dict_serialization_function
},
...

will call a different function when serializing the value to each of the four formats given above.

Tip

If on_serialize is None, or if its value for a particular format is None, then SQLAthanor will default to a Default Serialization Function based on the data type of the model attribute.

If defining your own custom serializer function, please bear in mind that a valid serializer function will:

accept one positional argument, which is the value of the model attribute to be serialized, and

return one value, which is the value that will be included in the serialized output.

See also

De-serialization Post-processing ¶

The on_deserialize configuration argument allows you to assign a de-serialization function to a particular model attribute.

If assigned, the de-serialization function will be called after your serialized object is parsed, but before the value is assigned to your Python model attribute. This is particularly useful when you need to:

convert a value from its serialized format (e.g. a string) into the type supported by your model class (e.g. an integer),
validate that a serialized value is “correct” (matches your expectations),
do something to the value before persisting it (e.g. hash and salt) to the database.

A typical example of this might be:

converting an empty string '' in a CSV record into None
validating that the serialized value is a proper telephone number
hashing an inbound password.

The on_deserialize argument expects to receive either a callable (a Python function), or a dict where keys correspond to SQLAthanor’s supported formats and values are the callables to use for that format. Thus:

...
on_deserialize = my_deserialization_function,
...

will call the my_deserialization_function() whenever de-serializing the value, but

...
on_deserialize = {
    'csv': my_csv_deserialization_function,
    'json': my_json_deserialization_function,
    'yaml': my_yaml_deserialization_function,
    'dict': my_dict_deserialization_function
},
...

will call a different function when de-serializing the value to each of the four formats given above.

Tip

If on_deserialize is None, or if its value for a particular format is None, then SQLAthanor will default to a Default De-serialization Function based on the data type of the model attribute being de-serialized.

If defining your own custom deserializer function, please bear in mind that a valid deserializer function will:

accept one positional argument, which is the value for the model attribute as found in the serialized input, and

return one value, which is the value that will be assigned to the model attribute (and thus probably persisted to the underlying database).

See also

6. Serializing a Model Instance ¶

Once you’ve configured your model class, you can now easily serialize it to the formats you have enabled. Your model instance will have one serialization method for each of the formats, named to_<format> where <format> corresponds to csv, json, yaml, and dict:

to_csv()

to_json()

to_yaml()

to_dict()

Nesting Complex Data ¶

SQLAthanor automatically supports nesting complex structures in JSON, YAML, and dict. However, to prevent the risk of infinite recursion, those formats serialization methods all feature a required max_nesting argument. By default, it is set to 0 which prevents model attributes that resolve to another model class from being included in a serialized output.

Unlike other supported formats, CSV works best with “flat” structures where each output column contains one and only one simple value, and so to_csv() does not include any max_nesting or current_nesting arguments.

Tip

As a general rule of thumb, we recommend that you avoid enabling CSV serialization on relationships unless using a custom serialization function to structure nested data.

to_csv()¶

BaseModel.to_csv(include_header=False, delimiter='|', wrap_all_strings=False, null_text='None', wrapper_character="'", double_wrapper_character_when_nested=False, escape_character='\\', line_terminator='\r\n')[source]¶

Retrieve a CSV string with the object’s data.

Parameters:	include_header (`bool`) – If `True`, will include a header row with column labels. If `False`, will not include a header row. Defaults to `True`. delimiter (`str`) – The delimiter used between columns. Defaults to `\|`. wrap_all_strings (`bool`) – If `True`, wraps any string data in the `wrapper_character`. If `None`, only wraps string data if it contains the `delimiter`. Defaults to `False`. null_text (`str`) – The text value to use in place of empty values. Only applies if `wrap_empty_values` is `True`. Defaults to `'None'`. wrapper_character (`str`) – The string used to wrap string values when wrapping is necessary. Defaults to `'`. double_wrapper_character_when_nested (`bool`) – If `True`, will double the `wrapper_character` when it is found inside a column value. If `False`, will precede the `wrapper_character` by the `escape_character` when it is found inside a column value. Defaults to `False`. escape_character (`str`) – The character to use when escaping nested wrapper characters. Defaults to `\`. line_terminator (`str`) – The character used to mark the end of a line. Defaults to `\r\n`.
Returns:	Data from the object in CSV format ending in a newline (`\n`).
Return type:	`str`

to_json()¶

BaseModel.to_json(max_nesting=0, current_nesting=0, serialize_function=None, **kwargs)[source]¶

Return a JSON representation of the object.

Parameters:	max_nesting (`int`) – The maximum number of levels that the resulting JSON object can be nested. If set to `0`, will not nest other serializable objects. Defaults to `0`. current_nesting (`int`) – The current nesting level at which the `dict` representation will reside. Defaults to `0`. serialize_function (callable / `None`) – Optionally override the default JSON serializer. Defaults to `None`, which applies the default simplejson JSON serializer. Note Use the `serialize_function` parameter to override the default JSON serializer. A valid `serialize_function` is expected to accept a single `dict` and return a `str`, similar to `simplejson.dumps()`. If you wish to pass additional arguments to your `serialize_function` pass them as keyword arguments (in `kwargs`). kwargs (keyword arguments) – Optional keyword parameters that are passed to the JSON serializer function. By default, these are options which are passed to `simplejson.dumps()`.
Returns:	A `str` with the JSON representation of the object.
Return type:	`str`
Raises:	SerializableAttributeError – if attributes is empty MaximumNestingExceededError – if `current_nesting` is greater than `max_nesting` MaximumNestingExceededWarning – if an attribute requires nesting beyond `max_nesting`

to_yaml()¶

BaseModel.to_yaml(max_nesting=0, current_nesting=0, serialize_function=None, **kwargs)[source]¶

Return a YAML representation of the object.

Parameters:	max_nesting (`int`) – The maximum number of levels that the resulting object can be nested. If set to `0`, will not nest other serializable objects. Defaults to `0`. current_nesting (`int`) – The current nesting level at which the representation will reside. Defaults to `0`. serialize_function (callable / `None`) – Optionally override the default YAML serializer. Defaults to `None`, which calls the default `yaml.dump()` function from the PyYAML library. Note Use the `serialize_function` parameter to override the default YAML serializer. A valid `serialize_function` is expected to accept a single `dict` and return a `str`, similar to `yaml.dump()`. If you wish to pass additional arguments to your `serialize_function` pass them as keyword arguments (in `kwargs`). kwargs (keyword arguments) – Optional keyword parameters that are passed to the YAML serializer function. By default, these are options which are passed to `yaml.dump()`.
Returns:	A `str` with the JSON representation of the object.
Return type:	`str`
Raises:	SerializableAttributeError – if attributes is empty MaximumNestingExceededError – if `current_nesting` is greater than `max_nesting` MaximumNestingExceededWarning – if an attribute requires nesting beyond `max_nesting`

to_dict()¶

BaseModel.to_dict(max_nesting=0, current_nesting=0)[source]¶

Return a dict representation of the object.

Parameters:	max_nesting (`int`) – The maximum number of levels that the resulting `dict` object can be nested. If set to `0`, will not nest other serializable objects. Defaults to `0`. current_nesting (`int`) – The current nesting level at which the `dict` representation will reside. Defaults to `0`.
Returns:	A `dict` representation of the object.
Return type:	`dict`
Raises:	SerializableAttributeError – if attributes is empty MaximumNestingExceededError – if `current_nesting` is greater than `max_nesting` MaximumNestingExceededWarning – if an attribute requires nesting beyond `max_nesting`

7. Deserializing Data ¶

Once you’ve configured your model class, you can now easily de-serialize it from the formats you have enabled.

However, unlike serializing your data, there are actually two types of de-serialization method to choose from:

The new_from_<format> method operates on your model class directly and create a new model instance whose properties are set based on the data you are de-serializing.
The update_from_<format> methods operate on a model instance, and update that instance’s properties based on the data you are de-serializing.

Creating New:

new_from_csv()

new_from_json()

new_from_yaml()

new_from_dict()

Updating:

update_from_csv()

update_from_json()

update_from_yaml()

update_from_dict()

Creating New Instances ¶

new_from_csv()¶

classmethod BaseModel.new_from_csv(csv_data, delimiter='|', wrap_all_strings=False, null_text='None', wrapper_character="'", double_wrapper_character_when_nested=False, escape_character='\\', line_terminator='\r\n')[source]¶

Create a new model instance from a CSV record.

Tip

Unwrapped empty column values are automatically interpreted as null (None).

Parameters:	csv_data (`str`) – The CSV record. Should be a single row and should not include column headers. delimiter (`str`) – The delimiter used between columns. Defaults to `\|`. wrapper_character (`str`) – The string used to wrap string values when wrapping is applied. Defaults to `'`. null_text (`str`) – The string used to indicate an empty value if empty values are wrapped. Defaults to None.
Returns:	A model instance created from the record.
Return type:	model instance
Raises:	DeserializationError – if `csv_data` is not a valid `str` CSVStructureError – if the columns in `csv_data` do not match the expected columns returned by `get_csv_column_names()` ValueDeserializationError – if a value extracted from the CSV failed when executing its de-serialization function.

new_from_json()¶

classmethod BaseModel.new_from_json(input_data, deserialize_function=None, error_on_extra_keys=True, drop_extra_keys=False, **kwargs)[source]¶

Create a new model instance from data in JSON.

Parameters:

input_data (str) – The input JSON data.
deserialize_function (callable / None) –
Optionally override the default JSON deserializer. Defaults to None, which calls the default simplejson.loads() function from the doc:simplejson <simplejson:index> library.

Note

Use the deserialize_function parameter to override the default JSON deserializer.

A valid deserialize_function is expected to accept a single str and return a dict, similar to simplejson.loads().

If you wish to pass additional arguments to your deserialize_function pass them as keyword arguments (in kwargs).
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step passes the keys/values of the processed input data to your model’s __init__() method.

If your instance’s __init__() method does not support your extra keys, it will likely raise a TypeError.
drop_extra_keys (bool) – If True, will ignore unrecognized top-level keys in input_data. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.
kwargs (keyword arguments) – Optional keyword parameters that are passed to the JSON deserializer function. By default, these are options which are passed to simplejson.loads().

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a dict or JSON object serializable to a dict or if input_data is empty.

new_from_yaml()¶

classmethod BaseModel.new_from_yaml(input_data, deserialize_function=None, error_on_extra_keys=True, drop_extra_keys=False, **kwargs)[source]¶

Create a new model instance from data in YAML.

Parameters:

input_data (str) – The input YAML data.
deserialize_function (callable / None) –
Optionally override the default YAML deserializer. Defaults to None, which calls the default yaml.safe_load() function from the PyYAML library.

Note

Use the deserialize_function parameter to override the default YAML deserializer.

A valid deserialize_function is expected to accept a single str and return a dict, similar to yaml.safe_load().

If you wish to pass additional arguments to your deserialize_function pass them as keyword arguments (in kwargs).
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step passes the keys/values of the processed input data to your model’s __init__() method.

If your instance’s __init__() method does not support your extra keys, it will likely raise a TypeError.
drop_extra_keys (bool) – If True, will ignore unrecognized top-level keys in input_data. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a dict or JSON object serializable to a dict or if input_data is empty.

new_from_dict()¶

classmethod BaseModel.new_from_dict(input_data, error_on_extra_keys=True, drop_extra_keys=False)[source]¶

Update the model instance from data in a dict object.

Parameters:

input_data (dict) – The input dict
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step passes the keys/values of the processed input data to your model’s __init__() method.

If your instance’s __init__() method does not support your extra keys, it will likely raise a TypeError.
drop_extra_keys (bool) – If True, will omit unrecognized top-level keys from the resulting dict. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a dict or JSON object serializable to a dict or if input_data is empty.

Updating Instances ¶

update_from_csv()¶

BaseModel.update_from_csv(csv_data, delimiter='|', wrap_all_strings=False, null_text='None', wrapper_character="'", double_wrapper_character_when_nested=False, escape_character='\\', line_terminator='\r\n')[source]¶

Update the model instance from a CSV record.

Tip

Unwrapped empty column values are automatically interpreted as null (None).

Parameters:

csv_data (str) – The CSV record. Should be a single row and should not include column headers.
delimiter (str) – The delimiter used between columns. Defaults to |.
wrapper_character (str) – The string used to wrap string values when wrapping is applied. Defaults to '.
null_text (str) – The string used to indicate an empty value if empty values are wrapped. Defaults to None.

Raises:

DeserializationError – if csv_data is not a valid str
CSVStructureError – if the columns in csv_data do not match the expected columns returned by get_csv_column_names()
ValueDeserializationError – if a value extracted from the CSV failed when executing its de-serialization function.

update_from_json()¶

BaseModel.update_from_json(input_data, deserialize_function=None, error_on_extra_keys=True, drop_extra_keys=False, **kwargs)[source]¶

Update the model instance from data in a JSON string.

Parameters:

input_data (str) – The JSON data to de-serialize.
deserialize_function (callable / None) –
Optionally override the default JSON deserializer. Defaults to None, which calls the default simplejson.loads() function from the simplejson library.

Note

Use the deserialize_function parameter to override the default JSON deserializer.

A valid deserialize_function is expected to accept a single str and return a dict, similar to simplejson.loads().

If you wish to pass additional arguments to your deserialize_function pass them as keyword arguments (in kwargs).
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step attempts to set an attribute on the model instance for every top-level key in the parsed/processed input data.

If there is an extra key that cannot be set as an attribute on your model instance, it will raise AttributeError.
drop_extra_keys (bool) – If True, will ignore unrecognized keys in the input data. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.
kwargs (keyword arguments) – Optional keyword parameters that are passed to the JSON deserializer function.By default, these are options which are passed to simplejson.loads().

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a str JSON de-serializable object to a dict or if input_data is empty.

update_from_yaml()¶

BaseModel.update_from_yaml(input_data, deserialize_function=None, error_on_extra_keys=True, drop_extra_keys=False, **kwargs)[source]¶

Update the model instance from data in a YAML string.

Parameters:

input_data (str) – The YAML data to de-serialize.
deserialize_function (callable / None) –
Optionally override the default YAML deserializer. Defaults to None, which calls the default yaml.safe_load() function from the PyYAML library.

Note

Use the deserialize_function parameter to override the default YAML deserializer.

A valid deserialize_function is expected to accept a single str and return a dict, similar to yaml.safe_load().

If you wish to pass additional arguments to your deserialize_function pass them as keyword arguments (in kwargs).
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step attempts to set an attribute on the model instance for every top-level key in the parsed/processed input data.

If there is an extra key that cannot be set as an attribute on your model instance, it will raise AttributeError.
drop_extra_keys (bool) – If True, will ignore unrecognized keys in the input data. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.
kwargs (keyword arguments) – Optional keyword parameters that are passed to the YAML deserializer function. By default, these are options which are passed to yaml.safe_load().

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a str YAML de-serializable object to a dict or if input_data is empty.

update_from_dict()¶

BaseModel.update_from_dict(input_data, error_on_extra_keys=True, drop_extra_keys=False)[source]¶

Update the model instance from data in a dict object.

Parameters:

input_data (dict) – The input dict
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step attempts to set an attribute on the model instance for every top-level key in the parsed/processed input data.

If there is an extra key that cannot be set as an attribute on your model instance, it will raise AttributeError.
drop_extra_keys (bool) – If True, will omit unrecognized top-level keys from the resulting dict. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a dict or JSON object serializable to a dict or if input_data is empty.

Using Declarative Reflection with SQLAthanor ¶

SQLAlchemy supports the use of reflection with the SQLAlchemy Declarative ORM.

This is a process where SQLAlchemy automatically constructs a Declarative model class based on what it reads from the table definition stored in your SQL database or a corresponding Table instance already defined and registered with a MetaData object.

See also

SQLAlchemy: Reflecting Database Objects
SQLAlchemy: Using Reflection with Declarative
Using Automap with SQLAthanor

SQLAthanor is also compatible with this pattern. In fact, it works just as you might expect. At a minimum:

from sqlathanor import declarative_base, Column, relationship, AttributeConfiguration

from sqlalchemy import create_engine, Integer, String, Table
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.ext.associationproxy import association_proxy

engine = create_engine('... ENGINE CONFIGURATION GOES HERE ...')
# NOTE: Because reflection relies on a specific SQLAlchemy Engine existing, presumably
# you would know how to configure / instantiate your database engine using SQLAlchemy.
# This is just here for the sake of completeness.

BaseModel = declarative_base()

class ReflectedUser(BaseModel):
  __table__ = Table('users',
                    BaseModel.metadata,
                    autoload = True,
                    autoload_with = engine)

will read the structure of your users table, and populate your ReflectedUser model class with model attributes that correspond to the table’s columns as defined in the underlying SQL table.

Caution

By design, SQLAlchemy’s reflection ONLY reflects Column definitions. It does NOT reflect relationships that you may otherwise model using SQLAlchemy.

Because the ReflectedUser class inherits from the SQLAthanor base model, it establishes the __serialization__ attribute, and the to_csv(), to_json(), to_yaml(), and to_dict() methods on the ReflectedUser class.

When working with a reflected model class, you can configure serialization/deserialization using either the declarative or meta approach as you normally would.

Warning

In the example above, if the database table named users already has a Table associated with it, ReflectedUser will inherit the Column definitions from the “original” Table object.

If those column definitions are defined using sqlathanor.schema.Column with declarative, their serialization/deserialization will also be reflected (inherited).

However, the ReflectedUser model class will NOT inherit any serialization/deserialization configuration defined using the meta approach.

Just as with standard SQLAlchemy reflection, you can override your Column definitions in your reflecting class (ReflectedUser), or add additional relationship model attributes, hybrid properties, or association proxies to the reflecting class.

Using Automap with SQLAthanor ¶

New in version 0.2.0.

Caution

Automap was introduced in SQLAlchemy v.0.9.1. If you are using SQLAthanor with SQLAlchemy v.0.9.0, then if you attempt to use automap_base() you will raise a SQLAlchemySupportError.

The Automap Extension is an incredibly useful tool for modeling existing databases with minimal effort. What it does is it reasd your existing database’s metadata and automatically constructs SQLAlchemy Declarative ORM model classes populated with your tables’ columns.

Neat, right? Saves a ton of effort.

Using SQLAthanor you can ensure that your automapped (automatically generated) models support serialization and de-serialization.

First, you need to create your automapped classes. This works just like in Automap, only you import automap_base() from SQLAthanor instead as shown below:

from sqlathanor.automap import automap_base
from sqlalchemy import create_engine

# Create your Automap Base
Base = automap_base()

engine = create_engine('... DATABASE CONNECTION GOES HERE ...')

# Prepare your automap base. This reads your database and creates your models.
Base.prepare(engine, reflect = True)

# And here you can create a "User" model class and an "Address" model class.
User = Base.classes.users
Address = Base.classes.addresses

In the example above, we create User and Address model classes which will be populated with the columns and relationships from the users and addresses tables in the database.

Both User and Address will have all of the standard SQLAthanor methods and functionality. BUT! They won’t have any serialization/de-serialization configured.

Before you start working with your models, you can configure their serialization/de-serialization using either a declarative approach using .set_attribute_serialization_config() or a meta approach by setting the __serialization__ attribute directly:

User.set_attribute_serialization_config('email_address',
                                        supports_csv = True,
                                        supports_json = True,
                                        supports_yaml = True,
                                        supports_dict = True)
User.set_attribute_serialization_config('password',
                                        supports_csv = (True, False),
                                        supports_json = (True, False),
                                        supports_yaml = (True, False),
                                        supports_dict = (True, False),
                                        on_deserialize = my_encryption_function)

User.__serialization__ = [
    {
        'name': 'email_address',
        'supports_csv': True,
        'supports_json': True,
        'supports_yaml': True,
        'supports_dict': True
    },
    {
        'name': 'password',
        'supports_csv': (True, False),
        'supports_json': (True, False),
        'supports_yaml': (True, False),
        'supports_dict': (True, False),
        'on_deserialize': my_encryption_function
    }
]

Both snippets of code above tell the User model to include users.email_address in both serialized output, and to expect it in de-serialized input. It also tells the User model to never serialize the users.password column, but to expect it in inbound data to de-serialize.

See also

SQLAlchemy: Automap Extension
Using Declarative Reflection with SQLAthanor

API Reference¶

Declarative ORM
Schema
- Column
- relationship()
Attribute Configuration
- AttributeConfiguration
- validate_serialization_config()
Flask-SQLAlchemy / Flask-SQLAthanor
- initialize_flask_sqlathanor()
- FlaskBaseModel
Automap
SQLAthanor Internals
- RelationshipProperty

Declarative ORM ¶

SQLAthanor provides a drop-in replacement for SQLAlchemy’s declarative_base() function and decorators, which can either be used as a base for your SQLAlchemy models directly or can be mixed into your SQLAlchemy models.

See also

It is BaseModel which exposes the methods and properties that enable serialization and de-serialization support.

For more information, please see:

Using SQLAthanor.

BaseModel ¶

class BaseModel(*args, **kwargs)[source]¶

Base class that establishes shared methods, attributes, and properties.

When constructing your ORM models, inherit from (or mixin) this class to add support for serialization and de-serialization.

Note

It is this class which adds SQLAthanor’s methods and properties to your SQLAlchemy model.

If your SQLAlchemy models do not inherit from this class, then they will not actually support serialization or de-serialization.

You can construct your declarative models using three approaches:

By inheriting or mixing in BaseModel directly as shown in the examples below.

from sqlathanor import BaseModel

# EXAMPLE 1: As a direct parent class for your model.
class MyModel(BaseModel):

    # Standard SQLAlchemy declarative model definition goes here.

# EXAMPLE 2: As a mixin parent for your model.
class MyBaseModel(object):

    # An existing base model that you have developed.

class MyModel(MyBaseModel, BaseModel):

    # Standard SQLAlchemy declarative model definition goes here.

By calling the declarative_base() function from SQLAthanor:

from sqlathanor import declarative_base

MyBaseModel = declarative_base()

By decorating your base model class with the @as_declarative decorator:

from sqlathanor import as_declarative

@as_declarative
class MyBaseModel(object):

    # Standard SQLAlchemy declarative model definition goes here.

classmethod does_support_serialization(attribute, from_csv=None, to_csv=None, from_json=None, to_json=None, from_yaml=None, to_yaml=None, from_dict=None, to_dict=None)[source]¶

Indicate whether attribute supports serialization/deserializtion.

Parameters:	attribute (`str`) – The name of the attribute whose serialization support should be confirmed. from_csv (`bool` / `None`) – If `True`, includes attribute names that can be de-serialized from CSV strings. If `False`, includes attribute names that cannot be de-serialized from CSV strings. If `None`, will not include attributes based on CSV de-serialization support (but may include them based on other parameters). Defaults to `None`. to_csv (`bool` / `None`) – If `True`, includes attribute names that can be serialized to CSV strings. If `False`, includes attribute names that cannot be serialized to CSV strings. If `None`, will not include attributes based on CSV serialization support (but may include them based on other parameters). Defaults to `None`. from_json (`bool` / `None`) – If `True`, includes attribute names that can be de-serialized from JSON strings. If `False`, includes attribute names that cannot be de-serialized from JSON strings. If `None`, will not include attributes based on JSON de-serialization support (but may include them based on other parameters). Defaults to `None`. to_json (`bool` / `None`) – If `True`, includes attribute names that can be serialized to JSON strings. If `False`, includes attribute names that cannot be serialized to JSON strings. If `None`, will not include attributes based on JSON serialization support (but may include them based on other parameters). Defaults to `None`. from_yaml (`bool` / `None`) – If `True`, includes attribute names that can be de-serialized from YAML strings. If `False`, includes attribute names that cannot be de-serialized from YAML strings. If `None`, will not include attributes based on YAML de-serialization support (but may include them based on other parameters). Defaults to `None`. to_yaml (`bool` / `None`) – If `True`, includes attribute names that can be serialized to YAML strings. If `False`, includes attribute names that cannot be serialized to YAML strings. If `None`, will not include attributes based on YAML serialization support (but may include them based on other parameters). Defaults to `None`. from_dict (`bool` / `None`) – If `True`, includes attribute names that can be de-serialized from `dict` objects. If `False`, includes attribute names that cannot be de-serialized from `dict` objects. If `None`, will not include attributes based on `dict` de-serialization support (but may include them based on other parameters). Defaults to `None`. to_dict – If `True`, includes attribute names that can be serialized to `dict` objects. If `False`, includes attribute names that cannot be serialized to `dict` objects. If `None`, will not include attributes based on `dict` serialization support (but may include them based on other parameters). Defaults to `None`.
Returns:	`True` if the attribute’s serialization support matches, `False` if not, and `None` if no serialization support was specified.
Return type:	`bool` / `None`
Raises:	AttributeError – if `attribute` is not present on the object

classmethod get_attribute_serialization_config(attribute)[source]¶

Retrieve the AttributeConfiguration for attribute.

Parameters:	attribute (`str`) – The attribute/column name whose serialization configuration should be returned.
Returns:	The `AttributeConfiguration` for `attribute`.
Return type:	`AttributeConfiguration`

classmethod get_csv_column_names(deserialize=True, serialize=True)[source]¶

Retrieve a list of CSV column names.

Parameters:	deserialize (`bool`) – If `True`, returns columns that support de-serialization. If `False`, returns columns that do not support deserialization. If `None`, does not take deserialization into account. Defaults to `True`. serialize (`bool`) – If `True`, returns columns that support serialization. If `False`, returns columns that do not support serialization. If `None`, does not take serialization into account. Defaults to `True`.
Returns:	List of CSV column names, sorted according to their configuration.
Return type:	`list` of `str`

get_csv_data(delimiter='|', wrap_all_strings=False, null_text='None', wrapper_character="'", double_wrapper_character_when_nested=False, escape_character='\\', line_terminator='\r\n')[source]¶

Return the CSV representation of the model instance (record).

Parameters:	delimiter (`str`) – The delimiter used between columns. Defaults to `\|`. wrap_all_strings (`bool`) – If `True`, wraps any string data in the `wrapper_character`. If `None`, only wraps string data if it contains the `delimiter`. Defaults to `False`. null_text (`str`) – The text value to use in place of empty values. Only applies if `wrap_empty_values` is `True`. Defaults to `'None'`. wrapper_character (`str`) – The string used to wrap string values when wrapping is necessary. Defaults to `'`. double_wrapper_character_when_nested (`bool`) – If `True`, will double the `wrapper_character` when it is found inside a column value. If `False`, will precede the `wrapper_character` by the `escape_character` when it is found inside a column value. Defaults to `False`. escape_character (`str`) – The character to use when escaping nested wrapper characters. Defaults to `\`. line_terminator (`str`) – The character used to mark the end of a line. Defaults to `\r\n`.
Returns:	Data from the object in CSV format ending in `line_terminator`.
Return type:	`str`

classmethod get_csv_header(deserialize=None, serialize=True, delimiter='|', wrap_all_strings=False, null_text='None', wrapper_character="'", double_wrapper_character_when_nested=False, escape_character='\\', line_terminator='\r\n')[source]¶

Retrieve a header string for a CSV representation of the model.

Parameters:	delimiter (`str`) – The character(s) to utilize between columns. Defaults to a pipe (`\|`). wrap_all_strings (`bool`) – If `True`, wraps any string data in the `wrapper_character`. If `None`, only wraps string data if it contains the `delimiter`. Defaults to `False`. null_text (`str`) – The text value to use in place of empty values. Only applies if `wrap_empty_values` is `True`. Defaults to `'None'`. wrapper_character (`str`) – The string used to wrap string values when wrapping is necessary. Defaults to `'`. double_wrapper_character_when_nested (`bool`) – If `True`, will double the `wrapper_character` when it is found inside a column value. If `False`, will precede the `wrapper_character` by the `escape_character` when it is found inside a column value. Defaults to `False`. escape_character (`str`) – The character to use when escaping nested wrapper characters. Defaults to `\`. line_terminator (`str`) – The character used to mark the end of a line. Defaults to `\r\n`.
Returns:	A string ending in `line_terminator` with the model’s CSV column names listed, separated by the `delimiter`.
Return type:	`str`

classmethod get_csv_serialization_config(deserialize=True, serialize=True)[source]¶

Retrieve the CSV serialization configurations that apply for this object.

Parameters:	deserialize (`bool` / `None`) – If `True`, returns configurations for attributes that can be de-serialized from CSV strings. If `False`, returns configurations for attributes that cannot be de-serialized from CSV strings. If `None`, ignores de-serialization configuration when determining which attribute configurations to return. Defaults to `None`. serialize (`bool` / `None`) – If `True`, returns configurations for attributes that can be serialized to CSV strings. If `False`, returns configurations for attributes that cannot be serialized to CSV strings. If `None`, ignores serialization configuration when determining which attribute configurations to return. Defaults to `None`.
Returns:	Set of attribute serialization configurations that match the arguments supplied.
Return type:	`list` of `AttributeConfiguration`

classmethod get_dict_serialization_config(deserialize=True, serialize=True)[source]¶

Retrieve the dict serialization configurations that apply for this object.

Parameters:	deserialize (`bool` / `None`) – If `True`, returns configurations for attributes that can be de-serialized from `dict` objects. If `False`, returns configurations for attributes that cannot be de-serialized from `dict` objects. If `None`, ignores de-serialization configuration when determining which attribute configurations to return. Defaults to `None`. serialize (`bool` / `None`) – If `True`, returns configurations for attributes that can be serialized to `dict` objects. If `False`, returns configurations for attributes that cannot be serialized to `dict` objects. If `None`, ignores serialization configuration when determining which attribute configurations to return. Defaults to `None`.
Returns:	Set of attribute serialization configurations that match the arguments supplied.
Return type:	`list` of `AttributeConfiguration`

classmethod get_json_serialization_config(deserialize=True, serialize=True)[source]¶

Retrieve the JSON serialization configurations that apply for this object.

Parameters:	deserialize (`bool` / `None`) – If `True`, returns configurations for attributes that can be de-serialized from JSON strings. If `False`, returns configurations for attributes that cannot be de-serialized from JSON strings. If `None`, ignores de-serialization configuration when determining which attribute configurations to return. Defaults to `None`. serialize (`bool` / `None`) – If `True`, returns configurations for attributes that can be serialized to JSON strings. If `False`, returns configurations for attributes that cannot be serialized to JSON strings. If `None`, ignores serialization configuration when determining which attribute configurations to return. Defaults to `None`.
Returns:	Set of attribute serialization configurations that match the arguments supplied.
Return type:	`list` of `AttributeConfiguration`

classmethod get_primary_key_column_names()[source]¶

Retrieve the column names for the model’s primary key columns.

Return type:	`list` of `str`

classmethod get_primary_key_columns()[source]¶

Retrieve the model’s primary key columns.

Returns:	`list` of `Column` objects corresponding to the table’s primary key(s).
Return type:	`list` of `Column`

classmethod get_serialization_config(from_csv=None, to_csv=None, from_json=None, to_json=None, from_yaml=None, to_yaml=None, from_dict=None, to_dict=None, exclude_private=True)[source]¶

Retrieve a list of AttributeConfiguration objects corresponding to attributes whose values can be serialized from/to CSV, JSON, YAML, etc.

Parameters:	from_csv (`bool` / `None`) – If `True`, includes attribute names that can be de-serialized from CSV strings. If `False`, includes attribute names that cannot be de-serialized from CSV strings. If `None`, will not include attributes based on CSV de-serialization support (but may include them based on other parameters). Defaults to `None`. to_csv (`bool` / `None`) – If `True`, includes attribute names that can be serialized to CSV strings. If `False`, includes attribute names that cannot be serialized to CSV strings. If `None`, will not include attributes based on CSV serialization support (but may include them based on other parameters). Defaults to `None`. from_json (`bool` / `None`) – If `True`, includes attribute names that can be de-serialized from JSON strings. If `False`, includes attribute names that cannot be de-serialized from JSON strings. If `None`, will not include attributes based on JSON de-serialization support (but may include them based on other parameters). Defaults to `None`. to_json (`bool` / `None`) – If `True`, includes attribute names that can be serialized to JSON strings. If `False`, includes attribute names that cannot be serialized to JSON strings. If `None`, will not include attributes based on JSON serialization support (but may include them based on other parameters). Defaults to `None`. from_yaml (`bool` / `None`) – If `True`, includes attribute names that can be de-serialized from YAML strings. If `False`, includes attribute names that cannot be de-serialized from YAML strings. If `None`, will not include attributes based on YAML de-serialization support (but may include them based on other parameters). Defaults to `None`. to_yaml (`bool` / `None`) – If `True`, includes attribute names that can be serialized to YAML strings. If `False`, includes attribute names that cannot be serialized to YAML strings. If `None`, will not include attributes based on YAML serialization support (but may include them based on other parameters). Defaults to `None`. from_dict (`bool` / `None`) – If `True`, includes attribute names that can be de-serialized from `dict` objects. If `False`, includes attribute names that cannot be de-serialized from `dict` objects. If `None`, will not include attributes based on `dict` de-serialization support (but may include them based on other parameters). Defaults to `None`. to_dict – If `True`, includes attribute names that can be serialized to `dict` objects. If `False`, includes attribute names that cannot be serialized to `dict` objects. If `None`, will not include attributes based on `dict` serialization support (but may include them based on other parameters). Defaults to `None`. exclude_private (`bool`) – If `True`, will exclude private attributes whose names begin with a single underscore. Defaults to `True`.
Returns:	List of attribute configurations.
Return type:	`list` of `AttributeConfiguration`

classmethod get_yaml_serialization_config(deserialize=True, serialize=True)[source]¶

Retrieve the YAML serialization configurations that apply for this object.

Parameters:	deserialize (`bool` / `None`) – If `True`, returns configurations for attributes that can be de-serialized from YAML strings. If `False`, returns configurations for attributes that cannot be de-serialized from YAML strings. If `None`, ignores de-serialization configuration when determining which attribute configurations to return. Defaults to `None`. serialize (`bool` / `None`) – If `True`, returns configurations for attributes that can be serialized to YAML strings. If `False`, returns configurations for attributes that cannot be serialized to YAML strings. If `None`, ignores serialization configuration when determining which attribute configurations to return. Defaults to `None`.
Returns:	Set of attribute serialization configurations that match the arguments supplied.
Return type:	`list` of `AttributeConfiguration`

classmethod new_from_csv(csv_data, delimiter='|', wrap_all_strings=False, null_text='None', wrapper_character="'", double_wrapper_character_when_nested=False, escape_character='\\', line_terminator='\r\n')[source]¶

Create a new model instance from a CSV record.

Tip

Unwrapped empty column values are automatically interpreted as null (None).

Parameters:	csv_data (`str`) – The CSV record. Should be a single row and should not include column headers. delimiter (`str`) – The delimiter used between columns. Defaults to `\|`. wrapper_character (`str`) – The string used to wrap string values when wrapping is applied. Defaults to `'`. null_text (`str`) – The string used to indicate an empty value if empty values are wrapped. Defaults to None.
Returns:	A model instance created from the record.
Return type:	model instance
Raises:	DeserializationError – if `csv_data` is not a valid `str` CSVStructureError – if the columns in `csv_data` do not match the expected columns returned by `get_csv_column_names()` ValueDeserializationError – if a value extracted from the CSV failed when executing its de-serialization function.

classmethod new_from_dict(input_data, error_on_extra_keys=True, drop_extra_keys=False)[source]¶

Update the model instance from data in a dict object.

Parameters:

input_data (dict) – The input dict
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step passes the keys/values of the processed input data to your model’s __init__() method.

If your instance’s __init__() method does not support your extra keys, it will likely raise a TypeError.
drop_extra_keys (bool) – If True, will omit unrecognized top-level keys from the resulting dict. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a dict or JSON object serializable to a dict or if input_data is empty.

classmethod new_from_json(input_data, deserialize_function=None, error_on_extra_keys=True, drop_extra_keys=False, **kwargs)[source]¶

Create a new model instance from data in JSON.

Parameters:

input_data (str) – The input JSON data.
deserialize_function (callable / None) –
Optionally override the default JSON deserializer. Defaults to None, which calls the default simplejson.loads() function from the doc:simplejson <simplejson:index> library.

Note

Use the deserialize_function parameter to override the default JSON deserializer.

A valid deserialize_function is expected to accept a single str and return a dict, similar to simplejson.loads().

If you wish to pass additional arguments to your deserialize_function pass them as keyword arguments (in kwargs).
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step passes the keys/values of the processed input data to your model’s __init__() method.

If your instance’s __init__() method does not support your extra keys, it will likely raise a TypeError.
drop_extra_keys (bool) – If True, will ignore unrecognized top-level keys in input_data. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.
kwargs (keyword arguments) – Optional keyword parameters that are passed to the JSON deserializer function. By default, these are options which are passed to simplejson.loads().

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a dict or JSON object serializable to a dict or if input_data is empty.

classmethod new_from_yaml(input_data, deserialize_function=None, error_on_extra_keys=True, drop_extra_keys=False, **kwargs)[source]¶

Create a new model instance from data in YAML.

Parameters:

input_data (str) – The input YAML data.
deserialize_function (callable / None) –
Optionally override the default YAML deserializer. Defaults to None, which calls the default yaml.safe_load() function from the PyYAML library.

Note

Use the deserialize_function parameter to override the default YAML deserializer.

A valid deserialize_function is expected to accept a single str and return a dict, similar to yaml.safe_load().

If you wish to pass additional arguments to your deserialize_function pass them as keyword arguments (in kwargs).
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step passes the keys/values of the processed input data to your model’s __init__() method.

If your instance’s __init__() method does not support your extra keys, it will likely raise a TypeError.
drop_extra_keys (bool) – If True, will ignore unrecognized top-level keys in input_data. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a dict or JSON object serializable to a dict or if input_data is empty.

classmethod set_attribute_serialization_config(attribute, config=None, supports_csv=None, csv_sequence=None, supports_json=None, supports_yaml=None, supports_dict=None, on_deserialize=None, on_serialize=None)[source]¶

Set the serialization/de-serialization configuration for attribute.

Note

Supplying keyword arguments like supports_csv or supports_json will override any configuration set in config.

Parameters:

attribute (str) – The name of the model attribute whose serialization/de-serialization configuration is to be configured.
config (AttributeConfiguration / None) – The AttributeConfiguration to apply. If None, will set particular values based on their corresponding keyword arguments.
supports_csv (bool / tuple of form (inbound: bool, outbound: bool) / None) –
Determines whether the column can be serialized to or de-serialized from CSV format.

If True, can be serialized to CSV and de-serialized from CSV. If False, will not be included when serialized to CSV and will be ignored if present in a de-serialized CSV.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

If None, will retain whatever configuration is currently applied. Defaults to None
supports_json (bool / tuple of form (inbound: bool, outbound: bool) / None) –
Determines whether the column can be serialized to or de-serialized from JSON format.

If True, can be serialized to JSON and de-serialized from JSON. If False, will not be included when serialized to JSON and will be ignored if present in a de-serialized JSON.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

If None, will retain whatever configuration is currently applied. Defaults to None
supports_yaml (bool / tuple of form (inbound: bool, outbound: bool) / None) –
Determines whether the column can be serialized to or de-serialized from YAML format.

If True, can be serialized to YAML and de-serialized from YAML. If False, will not be included when serialized to YAML and will be ignored if present in a de-serialized YAML.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

If None, will retain whatever configuration is currently applied. Defaults to None
supports_dict (bool / tuple of form (inbound: bool, outbound: bool) / None) –
Determines whether the column can be serialized to or de-serialized to a Python dict.

If True, can be serialized to dict and de-serialized from a dict. If False, will not be included when serialized to dict and will be ignored if present in a de-serialized dict.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

If None, will retain whatever configuration is currently applied. Defaults to None
on_deserialize (callable / dict with formats as keys and values as callables / None) –
A function that will be called when attempting to assign a de-serialized value to the column. This is intended to either coerce the value being assigned to a form that is acceptable by the column, or raise an exception if it cannot be coerced.
Tip

If you need to execute different on_deserialize functions for different formats, you can also supply a dict:
```
on_deserialize = {
  'csv': csv_on_deserialize_callable,
  'json': json_on_deserialize_callable,
  'yaml': yaml_on_deserialize_callable,
  'dict': dict_on_deserialize_callable
}
```
If False, will clear the current configuration to apply the default.

If None, will retain whatever configuration is currently applied. Defaults to None
Tip

To clear the on_deserialize function, you can either supply a value of False or pass a dict with particular formats set to None:
```
on_deserialize = {
  'csv': None,
  'json': None,
  'yaml': None,
  'dict': None
}

# is equivalent to:

on_deserialize = False
```
This will revert the on_deserialize function to the attribute’s default on_deserialize function based on its data type.
on_serialize (callable / dict with formats as keys and values as callables / None / False) –
A function that will be called when attempting to serialize a value from the column.
Tip

If you need to execute different on_serialize functions for different formats, you can also supply a dict:
```
on_serialize = {
  'csv': csv_on_serialize_callable,
  'json': json_on_serialize_callable,
  'yaml': yaml_on_serialize_callable,
  'dict': dict_on_serialize_callable
}
```
If False, will clear the current configuration to apply the default configuration.

If None, will retain whatever configuration is currently applied. Defaults to None
Tip

To clear the on_serialize function, you need to pass False or a dict with particular formats set to None:
```
on_serialize = {
  'csv': None,
  'json': None,
  'yaml': None,
  'dict': None
}

# is equivalent to

on_serialize = False
```
This will revert the on_serialize function to the attribute’s default on_serialize function based on its data type.
csv_sequence (int / None / False) –
Indicates the numbered position that the column should be in in a valid CSV-version of the object.

Note

If not specified, the column will go after any columns that do have a csv_sequence assigned, sorted alphabetically.

If two columns have the same csv_sequence, they will be sorted alphabetically.

If False, will set the position to None <python:None>` which will position the column after any explicitly positioned columns in alphabetical order.

If None, will retain whatever configuration is currently applied. Defaults to None

Raises:

ValueError – if attribute does not match config.name if config is not None

to_csv(include_header=False, delimiter='|', wrap_all_strings=False, null_text='None', wrapper_character="'", double_wrapper_character_when_nested=False, escape_character='\\', line_terminator='\r\n')[source]¶

Retrieve a CSV string with the object’s data.

Parameters:	include_header (`bool`) – If `True`, will include a header row with column labels. If `False`, will not include a header row. Defaults to `True`. delimiter (`str`) – The delimiter used between columns. Defaults to `\|`. wrap_all_strings (`bool`) – If `True`, wraps any string data in the `wrapper_character`. If `None`, only wraps string data if it contains the `delimiter`. Defaults to `False`. null_text (`str`) – The text value to use in place of empty values. Only applies if `wrap_empty_values` is `True`. Defaults to `'None'`. wrapper_character (`str`) – The string used to wrap string values when wrapping is necessary. Defaults to `'`. double_wrapper_character_when_nested (`bool`) – If `True`, will double the `wrapper_character` when it is found inside a column value. If `False`, will precede the `wrapper_character` by the `escape_character` when it is found inside a column value. Defaults to `False`. escape_character (`str`) – The character to use when escaping nested wrapper characters. Defaults to `\`. line_terminator (`str`) – The character used to mark the end of a line. Defaults to `\r\n`.
Returns:	Data from the object in CSV format ending in a newline (`\n`).
Return type:	`str`

to_dict(max_nesting=0, current_nesting=0)[source]¶

Return a dict representation of the object.

Parameters:	max_nesting (`int`) – The maximum number of levels that the resulting `dict` object can be nested. If set to `0`, will not nest other serializable objects. Defaults to `0`. current_nesting (`int`) – The current nesting level at which the `dict` representation will reside. Defaults to `0`.
Returns:	A `dict` representation of the object.
Return type:	`dict`
Raises:	SerializableAttributeError – if attributes is empty MaximumNestingExceededError – if `current_nesting` is greater than `max_nesting` MaximumNestingExceededWarning – if an attribute requires nesting beyond `max_nesting`

to_json(max_nesting=0, current_nesting=0, serialize_function=None, **kwargs)[source]¶

Return a JSON representation of the object.

Parameters:	max_nesting (`int`) – The maximum number of levels that the resulting JSON object can be nested. If set to `0`, will not nest other serializable objects. Defaults to `0`. current_nesting (`int`) – The current nesting level at which the `dict` representation will reside. Defaults to `0`. serialize_function (callable / `None`) – Optionally override the default JSON serializer. Defaults to `None`, which applies the default simplejson JSON serializer. Note Use the `serialize_function` parameter to override the default JSON serializer. A valid `serialize_function` is expected to accept a single `dict` and return a `str`, similar to `simplejson.dumps()`. If you wish to pass additional arguments to your `serialize_function` pass them as keyword arguments (in `kwargs`). kwargs (keyword arguments) – Optional keyword parameters that are passed to the JSON serializer function. By default, these are options which are passed to `simplejson.dumps()`.
Returns:	A `str` with the JSON representation of the object.
Return type:	`str`
Raises:	SerializableAttributeError – if attributes is empty MaximumNestingExceededError – if `current_nesting` is greater than `max_nesting` MaximumNestingExceededWarning – if an attribute requires nesting beyond `max_nesting`

to_yaml(max_nesting=0, current_nesting=0, serialize_function=None, **kwargs)[source]¶

Return a YAML representation of the object.

Parameters:	max_nesting (`int`) – The maximum number of levels that the resulting object can be nested. If set to `0`, will not nest other serializable objects. Defaults to `0`. current_nesting (`int`) – The current nesting level at which the representation will reside. Defaults to `0`. serialize_function (callable / `None`) – Optionally override the default YAML serializer. Defaults to `None`, which calls the default `yaml.dump()` function from the PyYAML library. Note Use the `serialize_function` parameter to override the default YAML serializer. A valid `serialize_function` is expected to accept a single `dict` and return a `str`, similar to `yaml.dump()`. If you wish to pass additional arguments to your `serialize_function` pass them as keyword arguments (in `kwargs`). kwargs (keyword arguments) – Optional keyword parameters that are passed to the YAML serializer function. By default, these are options which are passed to `yaml.dump()`.
Returns:	A `str` with the JSON representation of the object.
Return type:	`str`
Raises:	SerializableAttributeError – if attributes is empty MaximumNestingExceededError – if `current_nesting` is greater than `max_nesting` MaximumNestingExceededWarning – if an attribute requires nesting beyond `max_nesting`

update_from_csv(csv_data, delimiter='|', wrap_all_strings=False, null_text='None', wrapper_character="'", double_wrapper_character_when_nested=False, escape_character='\\', line_terminator='\r\n')[source]¶

Update the model instance from a CSV record.

Tip

Unwrapped empty column values are automatically interpreted as null (None).

Parameters:

csv_data (str) – The CSV record. Should be a single row and should not include column headers.
delimiter (str) – The delimiter used between columns. Defaults to |.
wrapper_character (str) – The string used to wrap string values when wrapping is applied. Defaults to '.
null_text (str) – The string used to indicate an empty value if empty values are wrapped. Defaults to None.

Raises:

DeserializationError – if csv_data is not a valid str
CSVStructureError – if the columns in csv_data do not match the expected columns returned by get_csv_column_names()
ValueDeserializationError – if a value extracted from the CSV failed when executing its de-serialization function.

update_from_dict(input_data, error_on_extra_keys=True, drop_extra_keys=False)[source]¶

Update the model instance from data in a dict object.

Parameters:

input_data (dict) – The input dict
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step attempts to set an attribute on the model instance for every top-level key in the parsed/processed input data.

If there is an extra key that cannot be set as an attribute on your model instance, it will raise AttributeError.
drop_extra_keys (bool) – If True, will omit unrecognized top-level keys from the resulting dict. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a dict or JSON object serializable to a dict or if input_data is empty.

update_from_json(input_data, deserialize_function=None, error_on_extra_keys=True, drop_extra_keys=False, **kwargs)[source]¶

Update the model instance from data in a JSON string.

Parameters:

input_data (str) – The JSON data to de-serialize.
deserialize_function (callable / None) –
Optionally override the default JSON deserializer. Defaults to None, which calls the default simplejson.loads() function from the simplejson library.

Note

Use the deserialize_function parameter to override the default JSON deserializer.

A valid deserialize_function is expected to accept a single str and return a dict, similar to simplejson.loads().

If you wish to pass additional arguments to your deserialize_function pass them as keyword arguments (in kwargs).
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step attempts to set an attribute on the model instance for every top-level key in the parsed/processed input data.

If there is an extra key that cannot be set as an attribute on your model instance, it will raise AttributeError.
drop_extra_keys (bool) – If True, will ignore unrecognized keys in the input data. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.
kwargs (keyword arguments) – Optional keyword parameters that are passed to the JSON deserializer function.By default, these are options which are passed to simplejson.loads().

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a str JSON de-serializable object to a dict or if input_data is empty.

update_from_yaml(input_data, deserialize_function=None, error_on_extra_keys=True, drop_extra_keys=False, **kwargs)[source]¶

Update the model instance from data in a YAML string.

Parameters:

input_data (str) – The YAML data to de-serialize.
deserialize_function (callable / None) –
Optionally override the default YAML deserializer. Defaults to None, which calls the default yaml.safe_load() function from the PyYAML library.

Note

Use the deserialize_function parameter to override the default YAML deserializer.

A valid deserialize_function is expected to accept a single str and return a dict, similar to yaml.safe_load().

If you wish to pass additional arguments to your deserialize_function pass them as keyword arguments (in kwargs).
error_on_extra_keys (bool) –
If True, will raise an error if an unrecognized key is found in input_data. If False, will either drop or include the extra key in the result, as configured in the drop_extra_keys parameter. Defaults to True.

Warning

Be careful setting error_on_extra_keys to False.

This method’s last step attempts to set an attribute on the model instance for every top-level key in the parsed/processed input data.

If there is an extra key that cannot be set as an attribute on your model instance, it will raise AttributeError.
drop_extra_keys (bool) – If True, will ignore unrecognized keys in the input data. If False, will include unrecognized keys or raise an error based on the configuration of the error_on_extra_keys parameter. Defaults to False.
kwargs (keyword arguments) – Optional keyword parameters that are passed to the YAML deserializer function. By default, these are options which are passed to yaml.safe_load().

Raises:

ExtraKeyError – if error_on_extra_keys is True and input_data contains top-level keys that are not recognized as attributes for the instance model.
DeserializationError – if input_data is not a str YAML de-serializable object to a dict or if input_data is empty.

primary_key_value¶

The instance’s primary key.

Note

If not None, this value can always be passed to Query.get().

Warning

Returns None if the instance is pending, in a transient state, or does not have a primary key.

Returns:	scalar or `tuple` value representing the primary key. For a composite primary key, the order of identifiers corresponds to the order with which the model’s primary keys were defined. If no primary keys are available, will return `None`.
Return type:	scalar / `tuple` / `None`

declarative_base()¶

declarative_base(cls, **kwargs)[source]¶

Construct a base class for declarative class definitions.

The new base class will be given a metaclass that produces appropriate Table objects and makes the appropriate mapper calls based on the information provided declaratively in the class and any subclasses of the class.

Parameters:

cls (None / tuple of classes / class object) –
Defaults to BaseModel to provide serialization/de-serialization support.

If a tuple of classes, will include BaseModel in that list of classes to mixin serialization/de-serialization support.

If not None and not a tuple, will mixin BaseModel with the value passed to provide serialization/de-serialization support.
kwargs (keyword arguments) – Additional keyword arguments supported by the original sqlalchemy.ext.declarative.declarative_base() function

Returns:

Base class for declarative class definitions with support for serialization and de-serialization.

@as_declarative ¶

as_declarative(**kw)[source]¶

Class decorator for declarative_base().

Provides a syntactical shortcut to the cls argument sent to declarative_base(), allowing the base class to be converted in-place to a “declarative” base:

from sqlathanor import as_declarative

@as_declarative()
class Base(object):
    @declared_attr
    def __tablename__(cls):
        return cls.__name__.lower()

    id = Column(Integer,
                primary_key = True,
                supports_csv = True)

class MyMappedClass(Base):
    # ...

Tip

All keyword arguments passed to as_declarative() are passed along to declarative_base().

See also

declarative_base()

Schema ¶

The classes defined in sqlathanor.schema are drop-in replacements for SQLAlchemy’s Core Schema elements.

They add serialization and de-serialization support to your model’s columns and relationsips, and so should replace your imports of their SQLAlchemy analogs.

See also

Using SQLAthanor

Column ¶

class Column(*args, **kwargs)[source]¶

Represents a column in a database table. Inherits from sqlalchemy.schema.Column

__init__(*args, **kwargs)[source]¶

Construct a new Column object.

Warning

This method is analogous to the original SQLAlchemy Column.__init__() from which it inherits. The only difference is that it supports additional keyword arguments which are not supported in the original, and which are documented below.

For the original SQLAlchemy version, see:

SQLAlchemy: Column.__init__()

Parameters:

supports_csv (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from CSV format.

If True, can be serialized to CSV and de-serialized from CSV. If False, will not be included when serialized to CSV and will be ignored if present in a de-serialized CSV.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to CSV or de-serialized from CSV.
csv_sequence (int / None) –
Indicates the numbered position that the column should be in in a valid CSV-version of the object. Defaults to None.

Note

If not specified, the column will go after any columns that do have a csv_sequence assigned, sorted alphabetically.

If two columns have the same csv_sequence, they will be sorted alphabetically.
supports_json (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from JSON format.

If True, can be serialized to JSON and de-serialized from JSON. If False, will not be included when serialized to JSON and will be ignored if present in a de-serialized JSON.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to JSON or de-serialized from JSON.
supports_yaml (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from YAML format.

If True, can be serialized to YAML and de-serialized from YAML. If False, will not be included when serialized to YAML and will be ignored if present in a de-serialized YAML.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to YAML or de-serialized from YAML.
supports_dict (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized to a Python dict.

If True, can be serialized to dict and de-serialized from a dict. If False, will not be included when serialized to dict and will be ignored if present in a de-serialized dict.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to a dict or de-serialized from a dict.
on_deserialize (callable / dict with formats as keys and values as callables) –
A function that will be called when attempting to assign a de-serialized value to the column. This is intended to either coerce the value being assigned to a form that is acceptable by the column, or raise an exception if it cannot be coerced. If None, the data type’s default on_deserialize function will be called instead.
Tip

If you need to execute different on_deserialize functions for different formats, you can also supply a dict:
```
on_deserialize = {
  'csv': csv_on_deserialize_callable,
  'json': json_on_deserialize_callable,
  'yaml': yaml_on_deserialize_callable,
  'dict': dict_on_deserialize_callable
}
```
Defaults to None.
on_serialize (callable / dict with formats as keys and values as callables) –
A function that will be called when attempting to serialize a value from the column. If None, the data type’s default on_serialize function will be called instead.
Tip

If you need to execute different on_serialize functions for different formats, you can also supply a dict:
```
on_serialize = {
  'csv': csv_on_serialize_callable,
  'json': json_on_serialize_callable,
  'yaml': yaml_on_serialize_callable,
  'dict': dict_on_serialize_callable
}
```
Defaults to None.

relationship()¶

relationship(argument, supports_json = False, supports_yaml = False, supports_dict = False, **kwargs)¶

Provide a relationship between two mapped classes.

See also

Warning

This constructor is analogous to the original SQLAlchemy relationship() from which it inherits. The only difference is that it supports additional keyword arguments which are not supported in the original, and which are documented below.

For the original SQLAlchemy version, see: sqlalchemy.orm.relationship()

Parameters:

argument – see sqlalchemy.orm.relationship()
supports_json (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from JSON format.

If True, can be serialized to JSON and de-serialized from JSON. If False, will not be included when serialized to JSON and will be ignored if present in a de-serialized JSON.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to JSON or de-serialized from JSON.
supports_yaml (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from YAML format.

If True, can be serialized to YAML and de-serialized from YAML. If False, will not be included when serialized to YAML and will be ignored if present in a de-serialized YAML.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to YAML or de-serialized from YAML.
supports_dict (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized to a Python dict.

If True, can be serialized to dict and de-serialized from a dict. If False, will not be included when serialized to dict and will be ignored if present in a de-serialized dict.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to a dict or de-serialized from a dict.

Attribute Configuration ¶

The following classes and functions are used to apply meta configuration to your SQLAlchemy model.

See also

Using SQLAthanor

AttributeConfiguration ¶

class AttributeConfiguration(*args, **kwargs)[source]¶

Serialization/de-serialization configuration of a model attribute.

__init__(*args, **kwargs)[source]¶

Construct an AttributeConfiguration object.

Parameters:

name (str) – The name of the attribute. Defaults to None.
supports_csv (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from CSV format.

If True, can be serialized to CSV and de-serialized from CSV. If False, will not be included when serialized to CSV and will be ignored if present in a de-serialized CSV.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to CSV or de-serialized from CSV.
supports_json (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from JSON format.

If True, can be serialized to JSON and de-serialized from JSON. If False, will not be included when serialized to JSON and will be ignored if present in a de-serialized JSON.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to JSON or de-serialized from JSON.
supports_yaml (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized from YAML format.

If True, can be serialized to YAML and de-serialized from YAML. If False, will not be included when serialized to YAML and will be ignored if present in a de-serialized YAML.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to YAML or de-serialized from YAML.
supports_dict (bool / tuple of form (inbound: bool, outbound: bool)) –
Determines whether the column can be serialized to or de-serialized to a Python dict.

If True, can be serialized to dict and de-serialized from a dict. If False, will not be included when serialized to dict and will be ignored if present in a de-serialized dict.

Can also accept a 2-member tuple (inbound / outbound) which determines de-serialization and serialization support respectively.

Defaults to False, which means the column will not be serialized to a dict or de-serialized from a dict.
on_deserialize (callable / dict with formats as keys and values as callables) –
A function that will be called when attempting to assign a de-serialized value to the column. This is intended to either coerce the value being assigned to a form that is acceptable by the column, or raise an exception if it cannot be coerced. If None, the data type’s default on_deserialize function will be called instead.
Tip

If you need to execute different on_deserialize functions for different formats, you can also supply a dict:
```
on_deserialize = {
  'csv': csv_on_deserialize_callable,
  'json': json_on_deserialize_callable,
  'yaml': yaml_on_deserialize_callable,
  'dict': dict_on_deserialize_callable
}
```
Defaults to None.
on_serialize (callable / dict with formats as keys and values as callables) –
A function that will be called when attempting to serialize a value from the column. If None, the data type’s default on_serialize function will be called instead.
Tip

If you need to execute different on_serialize functions for different formats, you can also supply a dict:
```
on_serialize = {
  'csv': csv_on_serialize_callable,
  'json': json_on_serialize_callable,
  'yaml': yaml_on_serialize_callable,
  'dict': dict_on_serialize_callable
}
```
Defaults to None.
csv_sequence (int / None) –
Indicates the numbered position that the column should be in in a valid CSV-version of the object. Defaults to None.

Note

If not specified, the column will go after any columns that do have a csv_sequence assigned, sorted alphabetically.

If two columns have the same csv_sequence, they will be sorted alphabetically.
attribute (class attribute) – The object representation of an attribute. Supplying this value overrides any other configuration options supplied. Defaults to None.

classmethod from_attribute(attribute)[source]¶: Return an instance of AttributeConfiguration configured for a given attribute.

classmethod fromkeys(seq, value=None)[source]¶

Create a new AttributeConfiguration with keys in seq and values in value.

Parameters:	seq (iterable) – Iterable of keys value (iterable) – Iterable of values
Return type:	`AttributeConfiguration`

csv_sequence¶

Column position when serialized to or de-serialized from CSV.

Note

If None, will default to alphabetical sorting after any attributes that have an explicit csv_sequence provided.

Return type:	`int` / `None`.

name¶

The name of the attribute.

Return type:	`str` / `None`

on_deserialize¶

A function that will be called when attempting to assign a de-serialized value to the attribute.

This is intended to either coerce the value being assigned to a form that is acceptable by the attribute, or raise an exception if it cannot be coerced.

If None, the data type’s default on_deserialize function will be called instead.

Tip

If you need to execute different on_deserialize functions for different formats, you can also supply a dict:

on_deserialize = {
    'csv': csv_on_deserialize_callable,
    'json': json_on_deserialize_callable,
    'yaml': yaml_on_deserialize_callable,
    'dict': dict_on_deserialize_callable
}

Defaults to None.

Return type:	callable / `dict` with formats as keys and values as callables

on_serialize¶

A function that will be called when attempting to serialize a value from the attribute.

If None, the data type’s default on_serialize function will be called instead.

Tip

If you need to execute different on_serialize functions for different formats, you can also supply a dict:

on_serialize = {
    'csv': csv_on_serialize_callable,
    'json': json_on_serialize_callable,
    'yaml': yaml_on_serialize_callable,
    'dict': dict_on_serialize_callable
}

Defaults to None.

Return type:	callable / `dict` with formats as keys and values as callables

supports_csv¶

Indicates whether the attribute can be serialized / de-serialized to CSV.

Returns:	2-member `tuple` (inbound de-serialization / outbound serialization)
Return type:	`tuple` of form (`bool`, `bool`)

supports_dict¶

Indicates whether the attribute can be serialized / de-serialized to dict.

Returns:	2-member `tuple` (inbound de-serialization / outbound serialization)
Return type:	`tuple` of form (`bool`, `bool`)

supports_json¶

Indicates whether the attribute can be serialized / de-serialized to JSON.

Returns:	2-member `tuple` (inbound de-serialization / outbound serialization)
Return type:	`tuple` of form (`bool`, `bool`)

supports_yaml¶

Indicates whether the attribute can be serialized / de-serialized to YAML.

Returns:	2-member `tuple` (inbound de-serialization / outbound serialization)
Return type:	`tuple` of form (`bool`, `bool`)

validate_serialization_config()¶

validate_serialization_config(config)[source]¶

Validate that config contains AttributeConfiguration objects.

Parameters:	config (iterable of `AttributeConfiguration` objects / iterable of `dict` objects corresponding to a `AttributeConfiguration` / `AttributeConfiguration` / `dict` object corresponding to a `AttributeConfiguration`) – Object or iterable of objects that represent `AttributeConfigurations`
Return type:	`list` of `AttributeConfiguration` objects

Flask-SQLAlchemy / Flask-SQLAthanor ¶

initialize_flask_sqlathanor()¶

initialize_flask_sqlathanor(db)[source]¶

Initialize SQLAthanor contents on a Flask-SQLAlchemy instance.

Parameters:	db (`flask_sqlalchemy.SQLAlchemy`) – The `flask_sqlalchemy.SQLAlchemy` instance.
Returns:	A mutated instance of `db` that replaces SQLAlchemy components and their Flask-SQLAlchemy flavors with SQLAthanor analogs while maintaining Flask-SQLAlchemy and SQLAlchemy functionality and interfaces.
Return type:	`flask_sqlalchemy.SQLAlchemy`
Raises:	ImportError – if called when Flask-SQLAlchemy is not installed ValueError – if `db` is not an instance of `flask_sqlalchemy.SQLAlchemy`

FlaskBaseModel ¶

class FlaskBaseModel[source]¶

Base class that establishes shared methods, attributes, and properties.

Designed to be supplied as a model_class when initializing Flask-SQLAlchemy.

See also

For detailed explanation of functionality, please see BaseModel.

Automap ¶

automap_base(declarative_base=None, **kwargs)[source]¶

Produce a declarative automap base.

This function produces a new base class that is a product of the AutomapBase class as well as a declarative base that you supply.

If no declarative base is supplied, then the SQLAthanor default BaseModel will be used, to provide serialization/de-serialization support to the resulting automapped base class.

Parameters:	declarative_base (The declarative base model to combine with the automapped model class produced.) – The declarative base class that is to be combined with the `AutomapBase` class to construct the resulting automapped model class. To ensure that SQLAthanor serialization/de-serialization functionality is provided to your automapped model class, make sure that the value provided is produced by `sqlathanor.declarative_base()` or otherwise inherits from `sqlathanor.declarative.BaseModel`. If `None`, will default to `sqlathanor.declarative.BaseModel`. kwargs (keyword arguments) – Passed to `declarative_base()`
Returns:	A `AutomapBase` that can reflect your database schema structure with auto-generated declarative models that support SQLAthanor serialization/de-serialization.
Return type:	`AutomapBase`
Raises:	SQLAlchemySupportError – if called in an environment where SQLAlchemy is installed with a version less than 0.9.1 (which introduces automap support).

SQLAthanor Internals ¶

RelationshipProperty ¶

class RelationshipProperty(argument, supports_json=False, supports_yaml=False, supports_dict=False, on_serialize=None, on_deserialize=None, **kwargs)[source]¶

Describes an object property that holds a single item or list of items that correspond to a related database table.

Public constructor is the sqlathanor.schema.relationship() function.

Default Serialization Functions¶

See also

SQLAthanor applies default serialization functions to pre-process your model attributes before serializing them. These default functions primarily exist to convert a SQLAlchemy Data Type into a type appropriate for the format to which you are serializing your model instance.

The table below explains how a given model attribute data type is serialized using the default serialization function for each data type. If no information is provided, that means the value is serialized “as-is”.

Data Types	CSV	JSON	YAML	`dict`
`None`	Empty string `''`
`int` `Integer` `INTEGER` `INT` `long` `oracle.LONG` `mssql.TINYINT` `mysql.TINYINT` `mysql.MEDIUMINT` `SmallInteger` `SMALLINT` `BigInteger` `BIGINT` `mssql.UNIQUEIDENTIFIER`
`bool` `Boolean` `BOOLEAN`
`str` `String` `Text` `TEXT` `mssql.NTEXT` `mysql.LONGTEXT` `VARCHAR` `oracle.VARCHAR2` `NVARCHAR` `oracle.NVARCHAR2` `CHAR` `NCHAR` `Unicode` `UnicodeText` `CLOB` `oracle.NCLOB`
`float` `Float` `FLOAT` `decimal.Decimal` `DECIMAL` `complex` `REAL` `Numeric` `NUMERIC` `mysql.DOUBLE` `oracle.DOUBLE_PRECISION` `postgresql.DOUBLE_PRECISION`
`datetime.date` `Date` `DATE` `datetime.datetime` `DateTime` `DATETIME` `TIMESTAMP` `datetime.time` `Time` `TIME` `mssql.SMALLDATETIME` `mssql.DATETIME2`	ISO-8601 formatted string	ISO-8601 formatted string	ISO-8601 formatted string
`mysql.YEAR`
`bytes`	string	string	string
`datetime.timedelta` `Interval` `INTERVAL`	total number of seconds	total number of seconds	total number of seconds
`BINARY` `VARBINARY` `LargeBinary` `BLOB` `mysql.TINYBLOB` `mysql.MEDIUMBLOB` `mysql.LONGBLOB` `PickleType` `postgresql.BYTEA`		string
`mssql.MONEY` `mssql.SMALLMONEY` `postgresql.MONEY`
`oracle.BINARY_FLOAT` `oracle.BINARY_DOUBLE` `mssql.IMAGE`
`oracle.BFILE` `oracle.RAW`		string	string
`Enum` `ENUM` `SET` `postgresql.UUID` `mssql.BIT` `postgresql.BIT` `postgresql.REGCLASS` `postgresql.OID` `postgresql.CIDR` `postgresql.INET` `postgresql.MACADDR` `mssql.XML` `postgresql.TSVECTOR`		string	string	string
`dict` `list` `JSON` `postgresql.JSONB` `postgresql.HSTORE` `ARRAY`	`UnsupportedSerializationError`
`set` `tuple`	`UnsupportedSerializationError`	list	list
`mssql.SQL_VARIANT`	`UnsupportedSerializationError`	`UnsupportedSerializationError`	`UnsupportedSerializationError`	`UnsupportedSerializationError`

Default De-serialization Functions¶

See also

SQLAthanor applies default de-serialization functions to process your model attributes before assigning their de-serialized value to your model attribute. These default functions primarily exist to validate that a value assigned to a given attribute is valid given that attribute’s SQLAlchemy Data Type.

The table below shows the data type that is assigned to a model attribute by the default de-serialization function based on the attribute’s data type.

Note

If the default de-serializer function cannot coerce the value extracted from your serialized data to either None or the expected data type, SQLAthanor will raise ValueDeserializationError.

Data Types	CSV	JSON	YAML	`dict`
`None`
`int` `Integer` `INTEGER` `INT` `mssql.TINYINT` `mysql.TINYINT` `mysql.MEDIUMINT` `SmallInteger` `SMALLINT` `BigInteger` `BIGINT`	`int`	`int`	`int`	`int`
`bool` `Boolean` `BOOLEAN`	`bool`	`bool`	`bool`	`bool`
`str` `String` `Text` `TEXT` `mssql.NTEXT` `mysql.LONGTEXT` `VARCHAR` `oracle.VARCHAR2` `NVARCHAR` `oracle.NVARCHAR2` `CHAR` `NCHAR` `Unicode` `UnicodeText` `CLOB` `oracle.NCLOB` `Enum` `ENUM` `SET`	`str`	`str`	`str`	`str`
`float` `Float` `FLOAT` `oracle.BINARY_FLOAT`	`float`	`float`	`float`	`float`
`long` `oracle.LONG` `complex` `REAL` `Numeric` `NUMERIC` `mysql.DOUBLE` `oracle.DOUBLE_PRECISION` `postgresql.DOUBLE_PRECISION` `mssql.ROWVERSION` `oracle.BINARY_DOUBLE` `mssql.MONEY` `mssql.SMALLMONEY` `postgresql.MONEY`	`validator_collection.validators.numeric()`	`validator_collection.validators.numeric()`	`validator_collection.validators.numeric()`	`validator_collection.validators.numeric()`
`decimal.Decimal` `DECIMAL`	`decimal.Decimal`	`decimal.Decimal`	`decimal.Decimal`	`decimal.Decimal`
`datetime.date` `Date` `DATE`	`datetime.date`	`datetime.date`	`datetime.date`	`datetime.date`
`datetime.datetime` `DateTime` `DATETIME` `TIMESTAMP` `mssql.SMALLDATETIME` `mssql.DATETIME2`	`datetime.datetime`	`datetime.datetime`	`datetime.datetime`	`datetime.datetime`
`datetime.time` `Time` `TIME`	`datetime.time`	`datetime.time`	`datetime.time`	`datetime.time`
`mysql.YEAR`
`datetime.timedelta` `Interval` `INTERVAL`	`datetime.timedelta`	`datetime.timedelta`	`datetime.timedelta`	`datetime.timedelta`
`bytes` `mssql.IMAGE` `BINARY` `VARBINARY` `LargeBinary` `BLOB` `mysql.TINYBLOB` `mysql.MEDIUMBLOB` `mysql.LONGBLOB` `PickleType` `oracle.RAW`	`bytes`	`bytes`	`bytes`	`bytes`
`postgresql.BYTEA` `oracle.BFILE`
`postgresql.UUID`	`uuid.UUID`	`uuid.UUID`	`uuid.UUID`	`uuid.UUID`
`mssql.BIT` `postgresql.BIT`	`bool`	`bool`	`bool`	`bool`
`postgresql.REGCLASS` `postgresql.OID` `postgresql.CIDR` `postgresql.INET` `postgresql.TSVECTOR` `mssql.XML` `mssql.UNIQUEIDENTIFIER`
`postgresql.MACADDR`	`str`	`str`	`str`	`str`
`JSON` `postgresql.JSONB`	`UnsupportedDeserializationError`	`str`	`str`	`str`
`postgresql.HSTORE` `ARRAY`	`UnsupportedDeserializationError`
`dict` `list` `set` `tuple`	`UnsupportedDeserializationError`	iterable	iterable	iterable
`mssql.SQL_VARIANT`	`UnsupportedDeserializationError`	`UnsupportedDeserializationError`	`UnsupportedDeserializationError`	`UnsupportedDeserializationError`

Error Reference¶

Handling Errors
SQLAthanor Errors
SQLAthanor Warnings
- SQLAthanorWarning (from UserWarning)
- MaximumNestingExceededWarning (from SQLAthanorWarning)

Handling Errors ¶

Stack Traces ¶

Because SQLAthanor produces exceptions which inherit from the standard library, it leverages the same API for handling stack trace information. This means that it will be handled just like a normal exception in unit test frameworks, logging solutions, and other tools that might need that information.

Errors During Serialization ¶

See also

from sqlathanor.errors import SerializableAttributeError, \
  UnsupportedSerializationError, MaximumNestingExceededError

# For a SQLAlchemy Model Class named "User" and a model instance named "user".

try:
  as_csv = user.to_csv()
  as_json = user.to_json()
  as_yaml = user.to_yaml()
  as_dict = user.to_dict()
except SerializableAttributeError as error:
  # Handle the situation where "User" model class does not have any attributes
  # serializable to JSON.
  pass
except UnsupportedSerializationError as error:
  # Handle the situation where one of the "User" model attributes is of a data
  # type that does not support serialization.
  pass
except MaximumNestingExceededError as error:
  # Handle a situation where "user.to_json()" received max_nesting less than
  # current_nesting.
  #
  # This situation is typically an error on the programmer's part, since
  # SQLAthanor by default avoids this kind of situation.
  #
  # Best practice is simply to let this exception bubble up.
  raise error

Errors During De-Serialization ¶

See also

from sqlathanor.errors import DeserializableAttributeError, \
  CSVStructureError, DeserializationError, ValueDeserializationError, \
  ExtraKeysError, UnsupportedDeserializationError

# For a SQLAlchemy Model Class named "User" and a model instance named "user",
# with serialized data in "as_csv", "as_json", "as_yaml", and "as_dict" respectively.

try:
  user.update_from_csv(as_csv)
  user.update_from_json(as_json)
  user.update_from_yaml(as_yaml)
  user.update_from_dict(as_dict)

  new_user = User.new_from_csv(as_csv)
  new_user = User.new_from_json(as_json)
  new_user = User.new_from_yaml(as_yaml)
  new_user = User.new_from_dict(as_dict)
except DeserializableAttributeError as error:
  # Handle the situation where "User" model class does not have any attributes
  # de-serializable from the given format (CSV, JSON, YAML, or dict).
  pass
except DeserializationError as error:
  # Handle the situation where the serialized object ("as_csv", "as_json",
  # "as_yaml", "as_dict") cannot be parsed, for example because it is not
  # valid JSON, YAML, or dict.
  pass
except CSVStructureError as error:
  # Handle the situation where the structure of "as_csv" does not match the
  # expectation configured for the "User" model class.
  raise error
except ExtraKeysError as error:
  # Handle the situation where the serialized object ("as_json",
  # "as_yaml", "as_dict") may have unexpected keys/attributes and
  # the error_on_extra_keys argument is False.
  #
  # Applies to: *_from_json(), *_from_yaml(), and *_from_dict() methods
  pass
except ValueDeserializationError as error:
  # Handle the situation where an input value in the serialized object
  # raises an exception in the deserialization post-processing function.
  pass
except UnsupportedDeserializationError as error:
  # Handle the situation where the de-serialization process attempts to
  # assign a value to an attribute that does not support de-serialization.
  pass

SQLAthanor Errors ¶

SQLAthanorError (from `ValueError`)¶

class SQLAthanorError[source]¶: Base error raised by SQLAthanor. Inherits from ValueError.

SQLAlchemySupportError (from `SQLAthanorError`)¶

class SQLAlchemySupportError[source]¶: Error raised when attempting to leverage a SQLAlchemy functionality that is not supported in your environment’s version of SQLAlchemy.

InvalidFormatError (from `SQLAthanorError`)¶

class InvalidFormatError[source]¶: Error raised when supplying a format that is not recognized by SQLAthanor.

SerializationError (from `SQLAthanorError`)¶

class SerializationError[source]¶: Error raised when something went wrong during serialization.

ValueSerializationError (from `SerializationError`)¶

class ValueSerializationError[source]¶: Error raised when an attribute value fails the serialization process.

SerializableAttributeError (from `SerializationError`)¶

class SerializableAttributeError[source]¶: Error raised when there are no serializable attributes on a model instance.

MaximumNestingExceededError (from `SerializationError`)¶

class MaximumNestingExceededError[source]¶: Error raised when the maximum permitted nesting is exceeded during serialization.

UnsupportedSerializationError (from `SerializationError`)¶

class UnsupportedSerializationError[source]¶: Error raised when attempting to serialize an attribute that does not support serialization.

DeserializationError (from `SQLAthanorError`)¶

class DeserializationError[source]¶: Error raised when something went wrong during de-serialization.

CSVStructureError (from `DeserializationError`)¶

class CSVStructureError[source]¶: Error raised when there is a mismatch between expected columns and found columns in CSV data.

JSONParseError (from `DeserializationError`)¶

class JSONParseError[source]¶: Error raised when something went wrong parsing input JSON data.

YAMLParseError (from `DeserializationError`)¶

class YAMLParseError[source]¶: Error raised when something went wrong parsing input YAML data.

DeserializableAttributeError (from `DeserializationError`)¶

class DeserializableAttributeError[source]¶: Error raised when there are no de-serializable attributes on a model instance or an inbound data source is empty.

ValueDeserializationError (from `DeserializationError`)¶

class ValueDeserializationError[source]¶: Error raised when an attribute value fails the de-serialization process.

UnsupportedDeserializationError (from `DeserializationError`)¶

class UnsupportedDeserializationError[source]¶: Error raised when attempting to de-serialize an attribute that does not support de-serialization.

ExtraKeyError (from `DeserializationError`)¶

class ExtraKeyError[source]¶: Error raised when an inbound object being de-serialized has extra (unrecognized) keys.

SQLAthanor Warnings ¶

SQLAthanorWarning (from `UserWarning`)¶

class SQLAthanorWarning[source]¶: Base warning raised by SQLAthanor. Inherits from UserWarning.

MaximumNestingExceededWarning (from `SQLAthanorWarning`)¶

class MaximumNestingExceededWarning[source]¶: Warning raised when the maximum permitted nesting is exceeded during serialization.

Contributing to SQLAthanor¶

Note

As a general rule of thumb, SQLAthanor applies PEP 8 styling, with some important differences.

Branch	Unit Tests
latest
v.0.2.0
v.0.1.1
v.0.1.0
develop

One of my favorite ways of thinking about idiomatic design comes from a talk given by Luciano Ramalho at Pycon 2016 [5] where he listed traits of a Pythonic API as being:

don’t force [the user] to write boilerplate code
provide ready to use functions and objects
don’t force [the user] to subclass unless there’s a very good reason
include the batteries: make easy tasks easy
are simple to use but not simplistic: make hard tasks possible
leverage the Python data model to:
- provide objects that behave as you expect
- avoid boilerplate through introspection (reflection) and metaprogramming.

Contents:

Design Philosophy
Style Guide
Dependencies
Preparing Your Development Environment
Ideas and Feature Requests
Testing
Submitting Pull Requests
Building Documentation
References

Design Philosophy ¶

SQLAthanor is meant to be a “beautiful” and “usable” library. That means that it should offer an idiomatic API that:

works out of the box as intended,
minimizes “bootstrapping” to produce meaningful output, and
does not force users to understand how it does what it does.

In other words:

Users should simply be able to drive the car without looking at the engine.

Style Guide ¶

Basic Conventions ¶

Do not terminate lines with semicolons.
Line length should have a maximum of approximately 90 characters. If in doubt, make a longer line or break the line between clear concepts.
Each class should be contained in its own file.
If a file runs longer than 2,000 lines…it should probably be refactored and split.
All imports should occur at the top of the file.

Do not use single-line conditions:

# GOOD
if x:
  do_something()

# BAD
if x: do_something()

When testing if an object has a value, be sure to use if x is None: or if x is not None. Do not confuse this with if x: and if not x:.
Use the if x: construction for testing truthiness, and if not x: for testing falsiness. This is different from testing:
- if x is True:
- if x is False:
- if x is None:
As of right now, because we feel that it negatively impacts readability and is less-widely used in the community, we are not using type annotations.

Naming Conventions ¶

variable_name and not variableName or VariableName. Should be a noun that describes what information is contained in the variable. If a bool, preface with is_ or has_ or similar question-word that can be answered with a yes-or-no.
function_name and not function_name or functionName. Should be an imperative that describes what the function does (e.g. get_next_page).
CONSTANT_NAME and not constant_name or ConstantName.
ClassName and not class_name or Class_Name.

Design Conventions ¶

Functions at the module level can only be aware of objects either at a higher scope or singletons (which effectively have a higher scope).

Functions and methods can use one positional argument (other than self or cls) without a default value. Any other arguments must be keyword arguments with default value given.

def do_some_function(argument):
  # rest of function...

def do_some_function(first_arg,
                     second_arg = None,
                     third_arg = True):
  # rest of function ...

Functions and methods that accept values should start by validating their input, throwing exceptions as appropriate.
When defining a class, define all attributes in __init__.
When defining a class, start by defining its attributes and methods as private using a single-underscore prefix. Then, only once they’re implemented, decide if they should be public.

Don’t be afraid of the private attribute/public property/public setter pattern:

class SomeClass(object):
  def __init__(*args, **kwargs):
    self._private_attribute = None

  @property
  def private_attribute(self):
    # custom logic which  may override the default return

    return self._private_attribute

  @setter.private_attribute
  def private_attribute(self, value):
    # custom logic that creates modified_value

    self._private_attribute = modified_value

Separate a function or method’s final (or default) return from the rest of the code with a blank line (except for single-line functions/methods).

Documentation Conventions ¶

We are very big believers in documentation (maybe you can tell). To document SQLAthanor we rely on several tools:

Sphinx [1]¶

Sphinx [1] is used to organize the library’s documentation into this lovely readable format (which is also published to ReadTheDocs [2]). This documentation is written in reStructuredText [3] files which are stored in <project>/docs.

Tip

As a general rule of thumb, we try to apply the ReadTheDocs [2] own Documentation Style Guide [4] to our RST documentation.

Hint

To build the HTML documentation locally:

In a terminal, navigate to <project>/docs.
Execute make html.

When built locally, the HTML output of the documentation will be available at ./docs/_build/index.html.

Docstrings ¶

Docstrings are used to document the actual source code itself. When writing docstrings we adhere to the conventions outlined in PEP 257.

Dependencies ¶

Preparing Your Development Environment ¶

In order to prepare your local development environment, you should:

Fork the Git repository.
Clone your forked repository.
Set up a virtual environment (optional).
Install dependencies:

sqlathanor/ $ pip install -r requirements.txt

And you should be good to go!

Ideas and Feature Requests ¶

Check for open issues or create a new issue to start a discussion around a bug or feature idea.

Testing ¶

If you’ve added a new feature, we recommend you:

create local unit tests to verify that your feature works as expected, and

run local unit tests before you submit the pull request to make sure nothing else got broken by accident.

See also

For more information about the SQLAthanor testing approach please see: Testing SQLAthanor

Submitting Pull Requests ¶

After you have made changes that you think are ready to be included in the main library, submit a pull request on Github and one of our developers will review your changes. If they’re ready (meaning they’re well documented, pass unit tests, etc.) then they’ll be merged back into the main repository and slated for inclusion in the next release.

Building Documentation ¶

In order to build documentation locally, you can do so from the command line using:

sqlathanor/ $ cd docs
sqlathanor/docs $ make html

When the build process has finished, the HTML documentation will be locally available at:

sqlathanor/docs/_build/html/index.html

Note

Built documentation (the HTML) is not included in the project’s Git repository. If you need local documentation, you’ll need to build it.

References ¶

[1]	(1, 2) http://sphinx-doc.org

[2]	(1, 2) https://readthedocs.org

[3]	http://www.sphinx-doc.org/en/stable/rest.html

[4]	http://documentation-style-guide-sphinx.readthedocs.io/en/latest/style-guide.html

[5]	https://www.youtube.com/watch?v=k55d3ZUF3ZQ

Testing SQLAthanor ¶

Contents

Testing SQLAthanor

Testing Philosophy ¶

Note

Unit tests for SQLAthanor are written using pytest [1] and a comprehensive set of test automation are provided by tox [2].

There are many schools of thought when it comes to test design. When building SQLAthanor, we decided to focus on practicality. That means:

DRY is good, KISS is better. To avoid repetition, our test suite makes extensive use of fixtures, parametrization, and decorator-driven behavior. This minimizes the number of test functions that are nearly-identical. However, there are certain elements of code that are repeated in almost all test functions, as doing so will make future readability and maintenance of the test suite easier.

Coverage matters…kind of. We have documented the primary intended behavior of every function in the SQLAthanor library, and the most-likely failure modes that can be expected. At the time of writing, we have about 85% code coverage. Yes, yes: We know that is less than 100%. But there are edge cases which are almost impossible to bring about, based on confluences of factors in the wide world. Our goal is to test the key functionality, and as bugs are uncovered to add to the test functions as necessary.

Test Organization ¶

Each individual test module (e.g. test_validators.py) corresponds to a conceptual grouping of functionality. For example:

test_validators.py tests validator functions found in sqlathanor/_validators.py

Certain test modules are tightly coupled, as the behavior in one test module may have implications on the execution of tests in another. These test modules use a numbering convention to ensure that they are executed in their required order, so that test_1_NAME.py is always executed before test_2_NAME.py.

Configuring & Running Tests ¶

Installing with the Test Suite ¶

$ pip install sqlathanor[tests]

See also

When you create a local development environment, all dependencies for running and extending the test suite are installed.

Command-line Options ¶

SQLAthanor does not use any custom command-line options in its test suite.

Tip

For a full list of the CLI options, including the defaults available, try:

sqlathanor $ cd tests/
sqlathanor/tests/ $ pytest --help

Configuration File ¶

Because SQLAthanor has a very simple test suite, we have not prepared a pytest.ini configuration file.

Running Tests ¶

tests/ $ pytest

tests/ $ pytest tests/test_module.py

tests/ $ pytest tests/test_module.py -k 'test_my_test_function'

Skipping Tests ¶

Note

Because of the simplicity of SQLAthanor, the test suite does not currently support any test skipping.

Incremental Tests ¶

Note

The SQLAthanor test suite does support incremental testing using, however at the moment none of the tests designed rely on this functionality.

A variety of test functions are designed to test related functionality. As a result, they are designed to execute incrementally. In order to execute tests incrementally, they need to be defined as methods within a class that you decorate with the @pytest.mark.incremental decorator as shown below:

@pytest.mark.incremental
class TestIncremental(object):
    def test_function1(self):
        pass
    def test_modification(self):
        assert 0
    def test_modification2(self):
        pass

This class will execute the TestIncremental.test_function1() test, execute and fail on the TestIncremental.test_modification() test, and automatically fail TestIncremental.test_modification2() because of the .test_modification() failure.

To pass state between incremental tests, add a state argument to their method definitions. For example:

@pytest.mark.incremental
class TestIncremental(object):
    def test_function(self, state):
        state.is_logged_in = True
        assert state.is_logged_in = True
    def test_modification1(self, state):
        assert state.is_logged_in is True
        state.is_logged_in = False
        assert state.is_logged_in is False
    def test_modification2(self, state):
        assert state.is_logged_in is True

Given the example above, the third test (test_modification2) will fail because test_modification updated the value of state.is_logged_in.

Note

state is instantiated at the level of the entire test session (one run of the test suite). As a result, it can be affected by tests in other test modules.

[1]	https://docs.pytest.org/en/latest/

[2]	https://tox.readthedocs.io

Release History ¶

Contents

Release History

Release 0.2.0 ¶

#21: Added support for SQLAlchemy Automap Extension.
#27: Added support for programmatically modifying serialization/de-serialization configuration after model definition.

Release 0.1.1 ¶

#22: Added unit tests testing support for SQLAlchemy Declarative Reflection (#22).
#23: Added documentation for SQLAthanor usage with SQLAlchemy Declarative Reflection.
#24: Added documentation comparing/contrasting to alternative serialization/deserialization libraries.
Fixed project URLs in setup.py for display on PyPi.

Release 0.1.0 ¶

First public release

Glossary¶

Association Proxy

A concept in SQLAlchemy that is used to define a model attribute within one model class that acts as a proxy (mapping to) a model attribute on a different model class, related by way of a relationship.

See also

SQLAlchemy: association_proxy

Athanor

An alchemical furnace, sometimes called a piger henricus (slow henry), philosophical furnace, Furnace of Arcana, or Tower furnace. They were used by alchemists to apply uniform heat over an extended (weeks, even!) period of time and require limited maintenance / management.

Basically, these were alchemical slow cookers.

The term “athanor” is believed to derive from the Arabic al-tannoor (“bread oven”) which Califate-era alchemical texts describe as being used for slow, uniform alchemical disgestion in talismanic alchemy.

Comma-Separated Value (CSV)

A text-based data exchange format where data is represented in one row of text, with fields (columns) separated by a delimiter character (typically a comma , or pipe |).

Declarative Configuration

A way of configuring serialization and de-serialization for particular model attributes when defining those attributes on the model class using the SQLAlchemy Declarative ORM.

Tip

The Declarative Configuration approach does not support serialization or de-serialization for model attributes that are not Column or relationship().

If you want to support serialization on hybrid properties, association proxies, or instance attributes please use Meta Configuration.

See also

De-serialization

De-Serialization - as you can probably guess - is the reverse of serialization. It’s the process whereby data is received in one format (say a JSON string) and is converted into a Python object (a model instance) that you can more easily work with in your Python code.

Think of it this way: A web app written in JavaScript needs to ask your Python code to register a user. Your Python code will need to know that user’s details to register the user. So how does the web app deliver that information to your Python code? It’ll most typically send JSON - but your Python code will need to then de-serialize (translate) it from JSON into an object representation (your User object) that it can work with.

De-serialization Function

A function that is called when de-serializing a specific value. The function accepts a single positional argument (the value to be de-serialized), does whatever it needs to do to the value, and then returns the value that will be assigned to the appropriate model attribute.

Typical usages include value validation and hashing/salting/encryption. SQLAthanor applies a set of default de-serialization functions that apply for the data types supported by SQLAlchemy and its dialects.

See also

De-serialization Post-processing

Drop-in Replacement

A Python library that extends the functionality of an existing library by inheriting from (and extending or modifying) its original classes or replacing its original functions.

Hybrid Property

A concept in SQLAlchemy that is used to define a model attribute that is not directly represented in the model class’s underlying database table (i.e. the hybrid property is calculated/determined on-the-fly in your Python code when referenced).

See also

SQLAlchemy: hybrid_property

Instance Attribute

A model attribute that is only present within a model instance that is defined using Python’s built-in @property decorator.

JavaScript Object Notation (JSON)

A lightweight data-interchange format that has become the de facto standard for communication across internet-enabled APIs.

For a formal definition, please see the ECMA-404 Standard: JSON Data Interchange Syntax

Meta Configuration

A way of configuring serialization and de-serialization using a private model attribute labeled __serialization__.

Tip

Meta configuration is used to configure serialization/de-serialization for hybrid properties, association proxies, and regular (non-hybrid) Python properties.

See also

Model Attribute

A property or attribute that belongs to a model class or model instance. It will typically correspond to an underlying database column, relationship (foreign key constraint), hybrid property, or association proxy.

Serialization and De-serialization both operate on model attributes.

Model Class

A model class is a Python class that is used to instantiate model instances. It typically is constructed using the SQLAlchemy ORM.

A model class is composed of one or more model attributes which correspond to columns in an underlying SQL table. The SQLAlchemy ORM maps the model class to a corresponding Table object, which in turn describes the structure of the underlying SQL table.

Note

Throughout SQLAthanor we use the terms “model class” and “model” interchangably.

Model Instance

A model instance is an object representation of a database record in your Python code. Technically, it is an instance of a model class.

It stores and exposes the record’s data and (if you’re using a robust ORM like SQLAlchemy) exposes methods to modify that data.

Note

Throughout SQLAthanor we use the terms “model instance” and “record” interchangably.

Object Relational Mapper (ORM)

An Object Relational Mapper (ORM) is a software tool that makes it easier to write code that reads data from or writes data to a relational database.

Fundamentally, it maps a class in your code to the tables and columns in the underlying database so that you can work with that class, rather than worrying about how to construct multiple (often related!) records directly in SQL.

The SQLAlchemy ORM is one of the most powerful Python ORMs available, and also provides a great Declarative system that makes their super-powerful ORM incredibly easy to use.

Pickling

A process of serializing a Python object to a binary representation. Typically performed using the pickle module from the standard Python library, or an outside pickling library like dill.

Relationship

A connection between two database tables or their corresponding model classes defined using a foreign key constraint.

Serialization

Serialization is a process where a Python object (like a model instance) is converted into a different format, typically more suited to transmission to or interpretation by some other program.

Think of it this way: You’ve got a virtual representation of some information in your Python code. It’s an object that you can work with in your Python code. But how do you give that information to some other application (like a web app) written in JavaScript? You serialize (translate) it into a format that other language can understand.

Serialization Function

A function that is called when serializing a specific value. The function accepts a single positional argument (the model attribute value to serialize), does whatever it needs to do to the value, and then returns the value that will be included in the serialized output.

Typical usages include value format conversion. SQLAthanor applies a set of default serialization functions that apply for the data types supported by SQLAlchemy and its dialects.

See also

Serialization Pre-processing

YAML Ain’t a Markup Language (YAML)

YAML is a text-based data serialization format similar in some respects to JSON. For more information, please see the YAML 1.2 (3rd Edition) Specification.

Note

If we’re being absolutely formal, JSON is actually a subset of YAML’s syntax. But that’s being needlessly formal.

SQLAthanor License¶

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

SQLAthanor is a Python library that extends SQLAlchemy’s fantastic Declarative ORM to provide easy-to-use record serialization/de-serialization with support for:

JSON

CSV

YAML

dict

The library works as a drop-in extension - change one line of existing code, and it should just work. Furthermore, it has been extensively tested on Python 2.7, 3.4, 3.5, and 3.6 using SQLAlchemy 0.9 and higher.

Contents

SQLAthanor

Installation ¶

To install SQLAthanor, just execute:

$ pip install sqlathanor

Dependencies ¶

Why SQLAthanor?¶

Odds are you’ve used SQLAlchemy before. And if you haven’t, why on earth not? It is hands down the best relational database toolkit and ORM available for Python, and has helped me quickly write code for many APIs, software platforms, and data science projects. Just look at some of these great features.

As its name suggests, SQLAlchemy focuses on the problem of connecting your Python code to an underlying relational (SQL) database. That’s a super hard problem, especially when you consider the complexity of abstraction, different SQL databases, different SQL dialects, performance optimization, etc. It ain’t easy, and the SQLAlchemy team has spent years building one of the most elegant solutions out there.

But as hard as Pythonically communicating with a database is, in the real world with microservices, serverless architectures, RESTful APIs and the like we often need to do more with the data than read or write from/to our database. In almost all of the projects I’ve worked on over the last two decades, I’ve had to:

hand data off in some fashion (serialize) for another program (possibly written by someone else in another programming language) to work with, or

accept and interpret data (de-serialize) received from some other program (possibly written by someone else in another programming language).

Python objects (pickled or not) are great, but they’re rarely the best way of transmitting data over the wire, or communicating data between independent applications. Which is where formats like JSON, CSV, and YAML come in.

So when writing many Python APIs, I found myself writing methods to convert my SQLAlchemy records (technically, model instances) into JSON or creating new SQLAlchemy records based on data I received in JSON. So after writing similar methods many times over, I figured a better approach would be to write the serialization/de-serialization code just once, and then re-use it across all of my various projects.

Which is how SQLAthanor came about.

It adds simple methods like to_json(), new_from_csv(), and update_from_csv() to your SQLAlchemy declarative models and provides powerful configuration options that give you tons of flexibility.

Key SQLAthanor Features ¶

Easy to adopt: Just tweak your existing SQLAlchemy import statements and you’re good to go.
With one method call, convert SQLAlchemy model instances to:
- CSV records
- JSON objects
- YAML objects
- dict objects
With one method call, create or update SQLAlchemy model instances from:
- dict objects
- CSV records
- JSON objects
- YAML objects
Decide which serialization formats you want to support for which models.
Decide which columns/attributes you want to include in their serialized form (and pick different columns for different formats, too).
Default validation for de-serialized data for every SQLAlchemy data type.
Customize the validation used when de-serializing particular columns to match your needs.
Works with Declarative Reflection and the Automap Extension.

SQLAthanor vs Alternatives ¶

Since serialization and de-serialization are common problems, there are a variety of alternative ways to serialize and de-serialize your SQLAlchemy models. Obviously, I’m biased in favor of SQLAthanor. But it might be helpful to compare SQLAthanor to some commonly-used alternatives:

Adding your own custom serialization/de-serialization logic to your SQLAlchemy declarative models is a very viable strategy. It’s what I did for years, until I got tired of repeating the same patterns over and over again, and decided to build SQLAthanor instead.

But of course, implementing custom serialization/de-serialization logic takes a bit of effort.

Tip

When to use it?

In practice, I find that rolling my own solution is great when it’s a simple model with very limited business logic. It’s a “quick-and-dirty” solution, where I’m trading rapid implementation (yay!) for less flexibility/functionality (boo!).

Considering how easy SQLAthanor is to configure / apply, however, I find that I never really roll my own serialization/de-serialization approach when working SQLAlchemy models any more.

The Marshmallow library and its Marshmallow-SQLAlchemy extension are both fantastic. However, they have one major architectural difference to SQLAthanor and several more minor differences:

The biggest difference is that by design, they force you to maintain two representations of your data model. One is your SQLAlchemy model class, while the other is your Marshmallow schema (which determines how your model is serialized/de-serialized). Marshmallow-SQLAlchemy specifically tries to simplify this by generating a schema based on your model class, but you still need to configure, manage, and maintain both representations - which as your project gets more complex, becomes non-trivial.

SQLAthanor by contrast lets you configure serialization/deserialization in your SQLAlchemy model class definition. You’re only maintaining one data model representation in your Python code, which is a massive time/effort/risk-reduction.

Other notable differences relate to the API/syntax used to support non-JSON formats. I think Marshmallow uses a non-obvious approach, while with SQLAthanor the APIs are clean and simple. Of course, on this point, YMMV.

Tip

When to use it?

Marshmallow has one advantage over SQLAthanor: It can serialize/de-serialize any Python object, whether it is a SQLAlchemy model class or not. SQLAthanor only works with SQLAlchemy.

As a result, it may be worth using Marshmallow instead of SQLAthanor if you expect to be serializing / de-serializing a lot of non-SQLAlchemy objects.

The Colander library and the ColanderAlchemy extension are both great, but they have a similar major architectural difference to SQLAthanor as Marshmallow/Marshmallow-SQLAlchemy:

By design, they force you to maintain two representations of your data model. One is your SQLAlchemy model class, while the other is your Colander schema (which determines how your model is serialized/de-serialized). ColanderAlchemy tries to simplify this by generating a schema based on your model class, but you still need to configure, manage, and maintain both representations - which as your project gets more complex, becomes non-trivial.

SQLAthanor by contrast lets you configure serialization/deserialization in your SQLAlchemy model class definition. You’re only maintaining one data model representation in your Python code, which is a massive time/effort/risk-reduction.

A second major difference is that, again by design, Colander is designed to serialize/de-serialize Python objects to a set of Python primitives. Since neither JSON, CSV, or YAML are Python primitives, you’ll still need to serialize/de-serialize Colander’s input/output to/from its final “transmissable” form. Once you’ve got a Python primitive, this isn’t difficult - but it is an extra step.

Tip

When to use it?

Colander has one advantage over SQLAthanor: It can serialize/de-serialize any Python object, whether it is a SQLAlchemy model class or not. SQLAthanor only works with SQLAlchemy.

As a result, it may be worth using Colander instead of SQLAthanor if you expect to be serializing / de-serializing a lot of non-SQLAlchemy objects.

pandas is one of my favorite analytical libraries. It has a number of great methods that adopt a simple syntax, like read_csv() or to_csv() which de-serialize / serialize data to various formats (including SQL, JSON, CSV, etc.).

So at first blush, one might think: Why not just use pandas to handle serialization/de-serialization?

Well, pandas isn’t really a serialization alternative to SQLAthanor. More properly, it is an ORM alternative to SQLAlchemy itself.

I could write (and have written) a lot on the subject, but the key difference is that pandas is a “lightweight” ORM that focuses on providing a Pythonic interface to work with the output of single SQL queries. It does not support complex relationships between tables, or support the abstracted definition of business logic that applies to an object representation of a “concept” stored in your database.

SQLAlchemy is specifically designed to do those things.

So you can think of pandas as being a less-abstract, “closer to bare metal” ORM - which is what you want if you want very efficient computations, on relatively “flat” (non-nested/minimally relational) data. Modification or manipulation of the data can be done by mutating your pandas DataFrame without too much maintenance burden because those mutations/modifications probably don’t rely too much on complex abstract business logic.

SQLAthanor piggybacks on SQLAlchemy’s business logic-focused ORM capabilities. It is designed to allow you to configure expected behavior once and then re-use that capability across all instances (records) of your data. And it’s designed to play well with all of the other complex abstractions that SQLAlchemy supports, like relationships, hybrid properties, reflection, or association proxies.

pandas serialization/de-serialization capabilities can only be configured “at use-time” (in the method call), which leads to a higher maintenance burden. SQLAthanor’s serialization/de-serialization capabilities are specifically designed to be configurable when defining your data model.

Tip

When to use it?

The decision of whether to use pandas or SQLAlchemy is a complex one, but in my experience a good rule of thumb is to ask yourself whether you’re going to need to apply complex business logic to your data.

The more complex the business logic is, the more likely SQLAlchemy will be a better solution. And if you are using SQLAlchemy, then SQLAthanor provides great and easy-to-use serialization/de-serialization capabilities.

Hello, World and Basic Usage ¶

SQLAthanor is a drop-in replacement for the SQLAlchemy Declarative ORM and parts of the SQLAlchemy Core.

1. Import SQLAthanor ¶

Since SQLAthanor is a drop-in replacement, you should import it using the same elements as you would import from SQLAlchemy:

The code below is a pretty standard set of import statements when working with SQLAlchemy and its Declarative ORM.

They’re provided for reference below, but do not make use of SQLAthanor and do not provide any support for serialization or de-serialization:

from sqlalchemy.ext.declarative import declarative_base, as_declarative
from sqlalchemy import Column, Integer, String          # ... and any other data types

# The following are optional, depending on how your data model is designed:
from sqlalchemy.orm import relationship
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.ext.associationproxy import association_proxy

To import SQLAthanor, just replace the relevant SQLAlchemy imports with their SQLAthanor counterparts as below:

from sqlathanor import declarative_base, as_declarative
from sqlathanor import Column
from sqlathanor import relationship             # This import is optional, depending on
                                                # how your data model is designed.

from sqlalchemy import Integer, String          # ... and any other data types

# The following are optional, depending on how your data model is designed:
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.ext.associationproxy import association_proxy

Tip

Because of its many moving parts, SQLAlchemy splits its various pieces into multiple modules and forces you to use many import statements.

The example above maintains this strategy to show how SQLAthanor is a 1:1 drop-in replacement. But obviously, you can import all of the items you need in just one import statement:

from sqlathanor import declarative_base, as_declarative, Column, relationship

SQLAthanor is designed to work with Flask-SQLAlchemy too! However, you need to:

Import the FlaskBaseModel class, and then supply it as the model_class argument when initializing Flask-SQLAlchemy.
Initialize SQLAthanor on your db instance using initialize_sqlathanor.

from sqlathanor import FlaskBaseModel, initialize_sqlathanor
from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy(model_class = FlaskBaseModel)
db = initialize_sqlathanor(db)

And that’s it! Now SQLAthanor serialization functionality will be supported by:

Flask-SQLAlchemy’s db.Model
Flask-SQLAlchemy’s db.relationship()
Flask-SQLAlchemy’s db.Column

See also

For more information about working with Flask-SQLAlchemy, please review their detailed documentation.

As the examples provided above show, importing SQLAthanor is very straightforward, and you can include it in an existing codebase quickly and easily. In fact, your code should work just as before. Only now it will include new functionality to support serialization and de-serialization.

The table below shows how SQLAlchemy classes and functions map to their SQLAthanor replacements:

SQLAlchemy Component	SQLAthanor Analog
`declarative_base()` from sqlalchemy.ext.declarative import declarative_base	`declarative_base()` from sqlathanor import declarative_base
`@as_declarative` from sqlalchemy.ext.declarative import as_declarative	`@as_declarative` from sqlathanor import as_declarative
`Column` from sqlalchemy import Column	`Column` from sqlathanor import Column
`relationship()` from sqlalchemy import relationship	`relationship()` from sqlathanor import relationship
`ext.automap.automap_base()` from sqlalchemy.ext.automap import automap_base	`automap.automap_base()` from sqlathanor.automap import automap_base

2. Declare Your Models ¶

Now that you have imported SQLAthanor, you can just declare your models the way you normally would, even using the exact same syntax.

But now when you define your model, you can also configure serialization and de-serialization for each attribute using two approaches:

The Meta Configuration approach lets you define a single __serialization__ attribute on your model that configures serialization/de-serialization for all of your model’s columns, hybrid properties, association proxies, and properties.

The Declarative Configuration approach lets you supply additional arguments to your attribute definitions that control whether and how they are serialized, de-serialized, or validated.

See also

Note

explicit is better than implicit

—PEP 20 - The Zen of Python

By default, all columns, relationships, association proxies, and hybrid properties will not be serialized. In order for a column, relationship, proxy, or hybrid property to be serializable to a given format or de-serializable from a given format, you will need to explicitly enable serialization/deserialization.

from sqlathanor import declarative_base, Column, relationship, AttributeConfiguration

from sqlalchemy import Integer, String
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.ext.associationproxy import association_proxy

BaseModel = declarative_base()

class User(BaseModel):
  __tablename__ = 'users'

  __serialization__ = [AttributeConfiguration(name = 'id',
                                              supports_csv = True,
                                              csv_sequence = 1,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None),
                       AttributeConfiguration(name = 'addresses',
                                              supports_json = True,
                                              supports_yaml = (True, True),
                                              supports_dict = (True, False),
                                              on_serialize = None,
                                              on_deserialize = None),
                       AttributeConfiguration(name = 'hybrid',
                                              supports_csv = True,
                                              csv_sequence = 2,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None)]
                       AttributeConfiguration(name = 'keywords',
                                              supports_csv = False,
                                              supports_json = True,
                                              supports_yaml = True,
                                              supports_dict = True,
                                              on_serialize = None,
                                              on_deserialize = None)]
                       AttributeConfiguration(name = 'python_property',
                                              supports_csv = (False, True),
                                              csv_sequence = 3,
                                              supports_json = (False, True),
                                              supports_yaml = (False, True),
                                              supports_dict = (False, True),
                                              on_serialize = None,
                                              on_deserialize = None)]

  id = Column('id',
              Integer,
              primary_key = True)

  addresses = relationship('Address',
                           backref = 'user')

  _hybrid = 1

  @hybrid_property
  def hybrid(self):
      return self._hybrid

  @hybrid.setter
  def hybrid(self, value):
      self._hybrid = value

  @hybrid.expression
  def hybrid(cls):
    return False

  keywords = association_proxy('keywords', 'keyword')

  @property
  def python_property(self):
    return self._hybrid * 2

As you can see, we’ve added a __serialization__ attribute to your standard model. The __serialization__ attribute takes a list of AttributeConfiguration instances, where each configures the serialization and de-serialization of a model attribute.

from sqlathanor import declarative_base

BaseModel = declarative_base()

class User(BaseModel):
  __tablename__ = 'users'

  id = Column("id",
              Integer,
              primary_key = True,
              autoincrement = True,
              supports_csv = True,
              csv_sequence = 1,
              supports_json = True,
              supports_yaml = True,
              supports_dict = True,
              on_serialize = None,
              on_deserialize = None)

  name = Column("name",
                Text,
                supports_csv = True,
                csv_sequence = 2,
                supports_json = True,
                supports_yaml = True,
                supports_dict = True,
                on_serialize = None,
                on_deserialize = None)

  email = Column("email",
                 Text,
                 supports_csv = True,
                 csv_sequence = 3,
                 supports_json = True,
                 supports_yaml = True,
                 supports_dict = True,
                 on_serialize = None,
                 on_deserialize = validators.email)

   password = Column("password",
                     Text,
                     supports_csv = (True, False),
                     csv_sequence = 4,
                     supports_json = (True, False),
                     supports_yaml = (True, False),
                     supports_dict = (True, False),
                     on_serialize = None,
                     on_deserialize = my_custom_password_hash_function)

As you can see, we’ve just added some (optional) arguments to the Column constructor. Hopefully these configuration arguments are self-explanatory.

Both the Meta and the Declarative configuration approaches use the same API for configuring serialization and de-serialization. While there are a lot of details, in general, the configuration arguments are:

supports_<format> determines whether that attribute is included when serializing or de-serializing the object to the <format> indicated.

Tip

If you give these options one value, it will either enable (True) or disable (False) both serialization and de-serialization, respectively.

But you can also supply a tuple with two values, where the first value controls whether the attribute supports the format when inbound (de-serialization) or whether it supports the format when outbound (serialization).

In the example above, the password attribute will not be included when serializing the object (outbound). But it will be expected / supported when de-serializing the object (inbound).
on_serialize indicates the function or functions that are used to prepare an attribute for serialization. This can either be a single function (that applies to all serialization formats) or a dict where each key corresponds to a format and its value is the function to use when serializing to that format.

Tip

If on_serialize is left as None, then SQLAthanor will apply a default on_serialize function based on the attribute’s data type.
on_deserialize indicates the function or functions that are used to validate or pre-process an attribute when de-serializing. This can either be a single function (that applies to all formats) or a dict where each key corresponds to a format and its value is the function to use when de-serializing from that format.

Tip

If on_deserialize is left as None, then SQLAthanor will apply a default on_deserialize function based on the attribute’s data type.

3. Serialize Your Model Instance ¶

See also

So now let’s say you have a model instance and want to serialize it. It’s super easy:

# Get user with id == 123 from the database
user = User.query.get(123)

# Serialize the user record to a JSON string.
serialized_version = user.to_json()

# Get user with id == 123 from the database
user = User.query.get(123)

# Serialize the user record to a CSV string.
serialized_version = user.to_csv()

# Get user with id == 123 from the database
user = User.query.get(123)

# Serialize the user record to a YAML string.
serialized_version = user.to_yaml()

# Get user with id == 123 from the database
user = User.query.get(123)

# Serialize the user record to a Python dict.
serialized_version = user.to_dict()

That’s it! Of course, the serialization methods all support a variety of other (optional!) options to fine-tune their behavior (CSV formatting, relationship nesting, etc.).

4. De-serialize a Model Instance ¶

See also

Now let’s say you receive a User object in serialized form and want to create a proper Python User object. That’s easy, too:

# EXAMPLE 1: Create a new User from a JSON string called "deserialized_object".
user = User.new_from_json(deserialized_object)

# EXAMPLE 2: Update an existing "user" instance from a JSON
# string called "deserialized_object".
user.update_from_json(updated_object)

# EXAMPLE 1: Create a new User from a CSV string called "deserialized_object".
user = User.new_from_csv(deserialized_object)

# EXAMPLE 2: Update an existing "user" instance from a CSV
# string called "deserialized_object".
user.update_from_csv(updated_object)

# EXAMPLE 1: Create a new User from a YAML string called "deserialized_object".
user = User.new_from_json(deserialized_object)

# EXAMPLE 2: Update an existing "user" instance from a YAML
# string called "deserialized_object".
user.update_from_yaml(updated_object)

# EXAMPLE 1: Create a new User from a dict called "deserialized_object".
user = User.new_from_dict(deserialized_object)

# EXAMPLE 2: Update an existing "user" instance from a dict called
# "deserialized_object".
user.update_from_dict(updated_object)

That’s it! Of course, all the de-serialization functions have additional options to fine-tune their behavior as needed. But that’s it.

Quickstart: Patterns and Best Practices¶

Using SQLAthanor¶

API Reference¶

Default Serialization Functions¶

Default De-serialization Functions¶

Error Reference¶

SQLAthanorError (from ValueError)¶

SQLAlchemySupportError (from SQLAthanorError)¶

InvalidFormatError (from SQLAthanorError)¶

SerializationError (from SQLAthanorError)¶

ValueSerializationError (from SerializationError)¶

SerializableAttributeError (from SerializationError)¶

MaximumNestingExceededError (from SerializationError)¶

UnsupportedSerializationError (from SerializationError)¶

DeserializationError (from SQLAthanorError)¶

CSVStructureError (from DeserializationError)¶

JSONParseError (from DeserializationError)¶

YAMLParseError (from DeserializationError)¶

DeserializableAttributeError (from DeserializationError)¶

ValueDeserializationError (from DeserializationError)¶

UnsupportedDeserializationError (from DeserializationError)¶

ExtraKeyError (from DeserializationError)¶

SQLAthanorWarning (from UserWarning)¶

MaximumNestingExceededWarning (from SQLAthanorWarning)¶

Contributing to SQLAthanor¶

Glossary¶

SQLAthanor License¶

SQLAthanorError (from `ValueError`)¶

SQLAlchemySupportError (from `SQLAthanorError`)¶

InvalidFormatError (from `SQLAthanorError`)¶

SerializationError (from `SQLAthanorError`)¶

ValueSerializationError (from `SerializationError`)¶

SerializableAttributeError (from `SerializationError`)¶

MaximumNestingExceededError (from `SerializationError`)¶

UnsupportedSerializationError (from `SerializationError`)¶

DeserializationError (from `SQLAthanorError`)¶

CSVStructureError (from `DeserializationError`)¶

JSONParseError (from `DeserializationError`)¶

YAMLParseError (from `DeserializationError`)¶

DeserializableAttributeError (from `DeserializationError`)¶

ValueDeserializationError (from `DeserializationError`)¶

UnsupportedDeserializationError (from `DeserializationError`)¶

ExtraKeyError (from `DeserializationError`)¶

SQLAthanorWarning (from `UserWarning`)¶

MaximumNestingExceededWarning (from `SQLAthanorWarning`)¶