Usage¶
Installation¶
ddbcereal is hosted on PyPI so it’s as simple as using your favorite installer:
python -m pip install ddbcereal
poetry add ddbcereal
The package uses semantic versioning to indicate backwards compatible changes to the API.
Communicating with DynamoDB¶
ddbcereal does not transport data to or from DynamoDB. It’s up to you to provide that layer. This is usually done with an AWS SDK or an HTTP library.
AWS SDKs¶
ddbcereal is known to work with these libraries:
aiobotocore
botocore
aioboto3’s low-level interface
boto3’s low-level interface
aiobotocore will be used for most examples in this documentation.
Raw HTTP API¶
ddbcereal can also work directly with the
DynamoDB HTTP API.
Before and after JSON processing, dicts can be fed to
Serializer
s and Deserializer
s constructed with
raw_transport=True
.
Basic Usage¶
Create a Serializer
to process data into the native DynamoDB format:
serializer.serialize(value)
to serialize individual valuesserializer.serialize_item(mapping)
to serialize an entire dict of values.
Create Deserializer
for getting DynamoDB data into native Python
values:
deserializer.deserialize(value)
to deserialize individual valuesdeserializer.deserialize_item(mapping)
for complete items from the AWS SDK
Serialize Python Data for DynamoDB¶
To prepare data for DynamoDB, construct a serializer object and use it to serialize items and attribute values as needed.
import aiobotocore
import ddbcereal
# Usually done at module level:
serializer = ddbcereal.Serializer()
aws = aiobotocore.Session()
async with aws.create_client('dynamodb') as ddb:
# For a given dict:
await ddb.put_item(
TableName='MyItems',
Item=serializer.serialize_item(my_dict)
)
# Adjust a single value:
await ddb.update_item(
TableName='Customers',
Key={'id': serializer.serialize(customer_id)},
UpdateExpression='SET displayName = :name, licenses = :licenses',
ExpressionAttributeValues={
':name': serializer.serialize('ACME, Inc.'),
':licenses': serializer.serialize(20)
}
)
Serializer Options¶
Serializers can be configured to handle data in different ways according to your needs.
- class Serializer(allow_inexact=False, validate_numbers=True, raw_transport=False, datetime_format=ddbcereal.ISO_8601, fraction_type=ddbcereal.NUMBER, empty_set_type=ddbcereal.NUMBER_SET)¶
- Parameters
allow_inexact (bool) – Whether to allow numbers whose exact value can’t be represented in DynamoDB or Python. DynamoDB’s Number type stores exact numbers (fixed decimals).
float
s are considered inexact by their nature and are only accepted with this option enabled.validate_numbers (bool) –
Whether to check inputted numbers to determine if they’re valid for storage in DynamoDB and whether or not they conform to the
allow_inexact
parameter.When enabled, attempts to serialize invalid numbers will result in a
ValueError
being raised. When disabled, serialization is faster, but mistakes might only be caught after the serialized value has been sent to DynamoDB.raw_transport (bool) – Indicates that values have not been pre-processed. For example, Base 64 strings have not been converted to bytes. Use this when using the AWS HTTP API without an AWS SDK.
datetime_format (DateFormat) – Determines how Python
datetime
s should be serialized. Possible enumerations are available on the ddbcereal top level module and theDateFormat
enum.fraction_type (DynamoDBType) – Determines how Python
Fraction
s should be serialized. Must beNUMBER
orSTRING
. Enumerations are available on the ddbcereal top level module and theDynamoDBType
enum.empty_set_type (DynamoDBType) – When an empty set is serialized, make the set this DynamoDB type. Must be
NUMBER_SET
,STRING_SET
, orBINARY_SET
. Enumerations are available on the ddbcereal top level module and theDynamoDBType
enum.
- class ddbcereal.DateFormat(value)¶
An enumeration.
- ISO_8601 = 3¶
DynamoDB String containing an ISO 8601 date string e.g. 2021-07-18T05:40:59.442117+00:00
- UNIX_MILLISECONDS = 2¶
DynamoDB Number holding number of milliseconds since the Unix epoch.
- UNIX_SECONDS = 1¶
DynamoDB Number holding number of seconds since the Unix epoch. This is the only date/time format that can used as a TTL field for automatic item expiration in DynamoDB.
Deserialize DynamoDB Data into Python¶
Construct a Deserializer
object and use it to deserialize items or
attribute values as needed.
import aiobotocore
import ddbcereal
deserializer = ddbcereal.Deserializer()
serializer = ddbcereal.Serializer()
aws = aiobotocore.Session()
async with aws.create_client('dynamodb') as ddb:
response = await ddb.query(
TableName='Companies',
KeyConditionExpression='id = :id',
ExpressionAttributeValues={
':id': serializer.serialize(target_id)
}
)
companies = [
deserializer.deserialize_item(item)
for item in response.get('Items', ())
]
process_companies(companies)
Deserializer Options¶
- class Deserializer(allow_inexact=False, raw_transport=False, number_type: PythonNumber = PythonNumber.DECIMAL_ONLY, null_value: Any = None, null_factory: Callable[[], Any] = None)¶
- Parameters
allow_inexact (bool) – Whether to allow conversion to a Python number that won’t exactly convey the value stored in DynamoDB (e.g. rounding of significant digits is required). Deserializing numbers to
float
s is only possible when this is enabled.raw_transport (bool) – Indicates to deserialize values to be transported without additional processing. Bytes will be transported as Base 64 strings. Use this when using the AWS HTTP API without an AWS SDK.
python_number (PythonNumber) –
Determines how DynamoDB Numbers should be serialized. Possible enumerations are available on the ddbcereal top level module and the
PythonNumber
enum:- class PythonNumber(value)¶
An enumeration.
- PythonNumber.DECIMAL_ONLY = 1¶
Only use decimal.Decimal. This is the default and the Python equivalent of DynamoDB Numbers.
- PythonNumber.FLOAT_ONLY = 2¶
Convert to Python float, losing accuracy if necessary.
- PythonNumber.FRACTION_ONLY = 3¶
Convert to fractions.Fraction. Maintains exactness.
- PythonNumber.INT_ONLY = 4¶
Only use Python ints. Will round if necessary if allow_inexact is True
- PythonNumber.INT_OR_DECIMAL = 5¶
Use int if the Number is whole, otherwise use decimal.Decimal
- PythonNumber.INT_OR_FLOAT = 6¶
Use int if the Number is whole, otherwise use a float, losing accuracy if necessary.
- PythonNumber.MOST_COMPACT = 7¶
Use int if the Number is whole, use float if a float’s decimal character representation matches the number, otherwise use a decimal.Decimal.
python_null_value – The Python value to convert DynamoDB Nulls to. Defaults to
None
. An immutable value is recommended. Ignored ifpython_null_factory
is supplied.python_null_factory (Callable[[], Any]) – A function invoked for every DynamoDB Null value. The Null is converted to the return value of the function.
python_null_value
is ignored if this is supplied.
Going Beyond the Basic Types¶
ddbcereal deserializers don’t know the final shape you want your data to conform to. They find appropriate Python types for the few types of data that DynamoDB can store. If you want to deserialize values into more advanced types, consider using a marshalling library like marshmallow or Pydantic.
They can take the dict produced by deserialize_item and create an objec based on a schema, an object with fields of built-in types like dates, deques and of custom types.
See
marshmallow.Schema.load()
and
Pydantic Models - Helper Functions.
Exceptions¶
- exception ddbcereal.NumberInexactError¶
Bases:
ValueError
A supplied number can’t be represented exactly by the target type and would either lose intent or data.
- exception ddbcereal.NumberNotAllowedError¶
Bases:
ValueError
A supplied number can’t be stored by DynamoDB.