core.db¶

Submodules¶

Attributes¶

`config`
`current_db_access_log`	If set to a set for the current thread/request, we'll log all entities / kinds accessed
`DATASTORE_BASE_TYPES`	Types that can be used in a datastore query
`KEY_SPECIAL_PROPERTY`	The property name pointing to an entities key in a query
`KeyType`	Alias that describes a key-type.

Classes¶

`Query`	Base Class for querying the datastore. Its API is similar to the google.cloud.datastore.query API,
`Entity`	The python representation of one datastore entity. The values of this entity are stored inside this dictionary,
`Key`	The python representation of one datastore key. Unlike the original implementation, we don't store a
`QueryDefinition`	A single Query that will be run against the datastore.
`SortOrder`	Create a collection of name/value pairs.

Functions¶

`allocate_ids`(kind_name[, num_ids, retry, timeout])
`AllocateIDs`(kind_name)	Allocates a new, free unique id for a given kind_name.
`count`([kind, up_to, queryDefinition])
`Count`([kind, up_to, queryDefinition])
`Delete`(keys)
`get`(keys)	Retrieves an entity (or a list thereof) from datastore.
`Get`(keys)
`put`(entities)	Save an entity in the Cloud Datastore.
`Put`(entities)
`run_in_transaction`(func, args, *kwargs)	Runs the function given in :param:callee inside a transaction.
`RunInTransaction`(callee, args, *kwargs)
`acquire_transaction_success_marker`()	Generates a token that will be written to the datastore (under "viur-transactionmarker") if the transaction
`encodeKey`(key)	Return the given key encoded as string (mimicking the old str() behaviour of keys)
`end_data_access_log`([outer_access_log])	Retrieves the set of entries accessed so far.
`endDataAccessLog`([outerAccessLog])
`fix_unindexable_properties`(entry, *[, keep_exclusions])	Recursively walk the given Entity and add all properties to the list of unindexed properties if they contain
`get_or_insert`(key, **kwargs)	Either creates a new entity with the given key, or returns the existing one.
`GetOrInsert`(key, **kwargs)
`is_in_transaction`()
`IsInTransaction`()
`key_helper`(in_key, target_kind[, ...])
`keyHelper`(inKey, targetKind[, additionalAllowedKinds, ...])
`normalize_key`(key)	Normalizes a datastore key (replacing _application with the current one)
`normalizeKey`(key)
`start_data_access_log`()	Clears our internal access log (which keeps track of which entries have been accessed in the current
`startDataAccessLog`()

Package Contents¶

core.db.config¶

class core.db.Query(kind, srcSkelClass=None, *args, **kwargs)¶

Bases: object

Base Class for querying the datastore. Its API is similar to the google.cloud.datastore.query API, but it provides the necessary hooks for relational or random queries, the fulltext search as well as support for IN filters.

Constructs a new Query. :param kind: The kind to run this query on. This may be later overridden to run on a different kind (like

viur-relations), but it’s guaranteed to return only entities of that kind.

Parameters:

srcSkelClass (Union[viur.core.skeleton.SkeletonInstance, None]) – If set, enables data-model depended queries (like relational queries) as well as the :meth:fetch method
kind (str)

kind¶

srcSkel = None¶

queries: None | core.db.types.QueryDefinition | List[core.db.types.QueryDefinition]¶

_filterHook: TFilterHook | None = None¶

_orderHook: TOrderHook | None = None¶

_customMultiQueryMerge: None | Callable[[Query, List[List[core.db.types.Entity]], int], List[core.db.types.Entity]] = None¶

_calculateInternalMultiQueryLimit: None | Callable[[Query, int], int] = None¶

customQueryInfo¶

origKind¶

_lastEntry = None¶

_fulltextQueryString: None | str = None¶

lastCursor = None¶

setFilterHook(hook)¶

Installs hook as a callback function for new filters.

hook will be called each time a new filter constrain is added to the query. This allows e.g. the relationalBone to rewrite constrains added after the initial processing of the query has been done (e.g. by listFilter() methods).

Parameters:: hook (TFilterHook) – The function to register as callback. A value of None removes the currently active hook.
Returns:: The previously registered hook (if any), or None.
Return type:: TFilterHook | None

setOrderHook(hook)¶

Installs hook as a callback function for new orderings.

hook will be called each time a db.Query.order() is called on this query.

Parameters:: hook (TOrderHook) – The function to register as callback. A value of None removes the currently active hook.
Returns:: The previously registered hook (if any), or None.
Return type:: TOrderHook | None

mergeExternalFilter(filters)¶

Safely merges filters according to the data model.

Its only valid to call this function if the query has been created using core.skeleton.Skeleton.all().

It’s safe to pass filters received from an external source (a user); unknown/invalid filters will be ignored, so the query-object is kept in a valid state even when processing malformed data.

If complex queries are needed (e.g. filter by relations), this function shall also be used.

See also filter() for simple filters.

Parameters:: filters (dict) – A dictionary of attributes and filter pairs.
Returns:: Returns the query itself for chaining.
Return type:: Self

filter(prop, value)¶

Adds a new constraint to this query.

See also mergeExternalFilter() for a safer filter implementation.

Parameters:

prop (str) – Name of the property + operation we’ll filter by
value (core.db.types.DATASTORE_BASE_TYPES | list[core.db.types.DATASTORE_BASE_TYPES]) – The value of that filter.

Returns:

Returns the query itself for chaining.

Return type:

Self

order(*orderings)¶

Specify a query sorting.

Resulting entities will be sorted by the first property argument, then by the second, and so on.

The following example

query = Query("Person")
query.order(("bday" db.SortOrder.Ascending), ("age", db.SortOrder.Descending))

sorts every Person in order of their birthday, starting with January 1. People with the same birthday are sorted by age, oldest to youngest.

order() may be called multiple times. Each call resets the sort order from scratch.

If an inequality filter exists in this Query it must be the first property passed to order(). t.Any number of sort orders may be used after the inequality filter property. Without inequality filters, any number of filters with different orders may be specified.

Entities with multiple values for an order property are sorted by their lowest value.

Note that a sort order implies an existence filter! In other words, Entities without the sort order property are filtered out, and not included in the query results.

If the sort order property has different types in different entities - e.g. if bob[“id”] is an int and fred[“id”] is a string - the entities will be grouped first by the property type, then sorted within type. No attempt is made to compare property values across types.

Parameters:: orderings (Tuple[str, core.db.types.SortOrder]) – The properties to sort by, in sort order. Each argument must be a (name, direction) 2-tuple.
Returns:: Returns the query itself for chaining.
Return type:: Self

setCursor(startCursor, endCursor=None)¶

Sets the start and optionally end cursor for this query.

The result set will only include results between these cursors. The cursor is generated by an earlier query with exactly the same configuration.

It’s safe to use client-supplied cursors, a cursor can’t be abused to access entities which don’t match the current filters.

Parameters:

startCursor (str) – The start cursor for this query.
endCursor (Optional[str]) – The end cursor for this query.

Returns:

Returns the query itself for chaining.

Return type:

Self

limit(limit)¶

Sets the query limit to limit entities in the result.

Parameters:: limit (int) – The maximum number of entities per batch.
Returns:: Returns the query itself for chaining.
Return type:: Self

distinctOn(keyList)¶

Ensure only entities with distinct values on the fields listed are returned. This will implicitly override your SortOrder as all fields listed in keyList have to be sorted first.

Parameters:: keyList (List[str])
Return type:: Self

getCursor()¶

Get a valid cursor from the last run of this query.

The source of this cursor varies depending on what the last call was: - run(): A cursor that points immediately behind the

last result pulled off the returned iterator.

get(): A cursor that points immediately behind the
last result in the returned list.

Returns:: A cursor that can be used in subsequent query requests or None if that query does not support cursors or there are no more elements to fetch
Return type:: Optional[str]

get_orders()¶

Get the orders from this query.

Returns:: The orders form this query as a list if there is no orders set it returns None
Return type:: List[Tuple[str, core.db.types.SortOrder]] | None

getKind()¶

Returns:: the current kind of this query. This may not be the kind this query has been constructed with as relational bones may rewrite this.
Return type:: str

_run_single_filter_query(query, limit)¶

Internal helper function that runs a single query definition on the datastore and returns a list of entities found. :param query: The querydefinition (filters, orders, distinct etc.) to run against the datastore :param limit: How many results should at most be returned :return: The first limit entities that matches this query

Parameters:

query (core.db.types.QueryDefinition)
limit (int)

Return type:

List[core.db.types.Entity]

_merge_multi_query_results(input_result)¶

Merge the lists of entries into a single list; removing duplicates and restoring sort-order :param input_result: Nested Lists of Entries returned by each individual query run :return: Sorted & deduplicated list of entries

Parameters:: input_result (List[List[core.db.types.Entity]])
Return type:: List[core.db.types.Entity]

_resort_result(entities, filters, orders)¶

Internal helper that takes a (deduplicated) list of entities that has been fetched from different internal queries (the datastore does not support IN filters itself, so we have to query each item in that array separately) and resorts the list so it matches the query again.

Parameters:

entities (List[core.db.types.Entity]) – t.List of entities to resort
filters (Dict[str, core.db.types.DATASTORE_BASE_TYPES]) – The filter used in the query (used to determine implicit sort order by an inequality filter)
orders (List[Tuple[str, core.db.types.SortOrder]]) – The sort-orders to apply

Returns:

The sorted list

Return type:

List[core.db.types.Entity]

_fixKind(resultList)¶

Jump to parentKind if necessary (used in relations)

Parameters:: resultList (List[core.db.types.Entity])
Return type:: List[core.db.types.Entity]

run(limit=-1)¶

Run this query.

It is more efficient to use limit if the number of results is known.

If queried data is wanted as instances of Skeletons, fetch() should be used.

Parameters:: limit (int) – Limits the query to the defined maximum entities.
Returns:: The list of found entities
Raises:: BadFilterError if a filter string is invalid
Raises:: BadValueError if a filter value is invalid.
Raises:: BadQueryError if an IN filter in combination with a sort order on another property is provided
Return type:: List[core.db.types.Entity]

count(up_to=2**63 - 1)¶

The count operation cost one entity read for up to 1,000 index entries matched (https://cloud.google.com/datastore/docs/aggregation-queries#pricing) :param up_to can be sigend int 64 bit (max positive 2^31-1)

Returns:: Count entries for this query.
Parameters:: up_to (int)
Return type:: int

fetch(limit=-1)¶

Run this query and fetch results as core.skeleton.SkelList.

This function is similar to run(), but returns a core.skeleton.SkelList instance instead of Entities.

Warning:: The query must be limited!
Parameters:: limit (int)
Return type:: viur.core.skeleton.SkelList

If queried data is wanted as instances of Entity, run() should be used.

Parameters:: limit (int) – Limits the query to the defined maximum entities.
Raises:: BadFilterError if a filter string is invalid
Raises:: BadValueError if a filter value is invalid.
Raises:: BadQueryError if an IN filter in combination with a sort order on another property is provided
Return type:: viur.core.skeleton.SkelList

iter()¶

Run this query and return an iterator for the results.

The advantage of this function is, that it allows for iterating over a large result-set, as it hasn’t have to be pulled in advance from the datastore.

This function intentionally ignores a limit set by limit().

Warning:: If iterating over a large result set, make sure the query supports cursors. Otherwise, it might not return all results as the AppEngine doesn’t maintain the view for a query for more than ~30 seconds.
Return type:: Iterator[core.db.types.Entity]

getEntry()¶

Returns only the first entity of the current query.

Returns:: The first entity on success, or None if the result-set is empty.
Return type:: Union[None, core.db.types.Entity]

getSkel()¶

Returns a matching core.db.skeleton.Skeleton instance for the current query.

It’s only possible to use this function if this query has been created using core.skeleton.Skeleton.all().

Returns:: The Skeleton or None if the result-set is empty.
Return type:: Optional[viur.core.skeleton.SkeletonInstance]

clone()¶

Returns a deep copy of the current query.

Returns:: The cloned query.
Return type:: Self

__repr__()¶

Return type:: str

core.db.allocate_ids(kind_name, num_ids=1, retry=None, timeout=None)¶

Parameters:

kind_name (str)
num_ids (int)

Return type:

list[core.db.types.Key]

core.db.AllocateIDs(kind_name)¶: Allocates a new, free unique id for a given kind_name.

core.db.count(kind=None, up_to=2**31 - 1, queryDefinition=None)¶

Parameters:

kind (str)
queryDefinition (core.db.types.QueryDefinition)

Return type:

int

core.db.Count(kind=None, up_to=2**31 - 1, queryDefinition=None)¶

Parameters:

kind (str)
queryDefinition (core.db.types.QueryDefinition)

Return type:

int

core.db.Delete(keys)¶

Parameters:: keys (Union[core.db.types.Entity, List[core.db.types.Entity], core.db.types.Key, List[core.db.types.Key]])

core.db.get(keys)¶

Retrieves an entity (or a list thereof) from datastore. If only a single key has been given we’ll return the entity or none in case the key has not been found, otherwise a list of all entities that have been looked up (which may be empty) :param keys: A datastore key (or a list thereof) to lookup :return: The entity (or None if it has not been found), or a list of entities.

Parameters:: keys (Union[core.db.types.Key, List[core.db.types.Key]])
Return type:: Union[List[core.db.types.Entity], core.db.types.Entity, None]

core.db.Get(keys)¶

Parameters:: keys (Union[core.db.types.Key, List[core.db.types.Key]])
Return type:: Union[List[core.db.types.Entity], core.db.types.Entity, None]

core.db.put(entities)¶

Save an entity in the Cloud Datastore. Also ensures that no string-key with a digit-only name can be used. :param entities: The entities to be saved to the datastore.

Parameters:: entities (Union[core.db.types.Entity, List[core.db.types.Entity]])

core.db.Put(entities)¶

Parameters:: entities (Union[core.db.types.Entity, List[core.db.types.Entity]])
Return type:: Union[core.db.types.Entity, None]

core.db.run_in_transaction(func, *args, **kwargs)¶

Runs the function given in :param:callee inside a transaction. Inside a transaction it’s guaranteed that - either all or no changes are written to the datastore - no other transaction is currently reading/writing the entities accessed

See (transactions)[https://cloud.google.com/datastore/docs/concepts/cloud-datastore-transactions] for more information.

..Warning: The datastore may produce unexpected results if an entity that have been written inside a transaction: is read (or returned in a query) again. In this case you will the the old state of that entity. Keep that in mind if wrapping functions to run in a transaction that may have not been designed to handle this case.

Parameters:

func (Callable) – The function that will be run inside a transaction
args – All args will be passed into the callee
kwargs – All kwargs will be passed into the callee

Returns:

Whatever the callee function returned

Raises:

RuntimeError – If the maximum transaction retries exceeded

Return type:

Any

core.db.RunInTransaction(callee, *args, **kwargs)¶

Parameters:: callee (Callable)
Return type:: Any

core.db.current_db_access_log: contextvars.ContextVar[set[Key | str] | None]¶: If set to a set for the current thread/request, we’ll log all entities / kinds accessed

core.db.DATASTORE_BASE_TYPES¶: Types that can be used in a datastore query

class core.db.Entity(key=None, exclude_from_indexes=None)¶

Bases: google.cloud.datastore.Entity

The python representation of one datastore entity. The values of this entity are stored inside this dictionary, while the meta-data (it’s key, the list of properties excluded from indexing and our version) as property values.

Parameters:

key (Optional[Key])
exclude_from_indexes (Optional[list[str]])

core.db.KEY_SPECIAL_PROPERTY = '__key__'¶: The property name pointing to an entities key in a query

class core.db.Key(*path_args, project=None, **kwargs)¶

Bases: google.cloud.datastore.Key

The python representation of one datastore key. Unlike the original implementation, we don’t store a reference to the project the key lives in. This is always expected to be the current project as ViUR does not support accessing data in multiple projects.

Parameters:: project (str | None)

__str__()¶

type core.db.KeyType = Key | str | int¶: Alias that describes a key-type.

class core.db.QueryDefinition¶

A single Query that will be run against the datastore.

kind: str | None¶: The datastore kind to run the query on. Can be None for kindles queries.

filters: TFilters¶: A dictionary of constrains to apply to the query.

orders: TOrders | None¶: The list of fields to sort the results by.

distinct: list[str] | None = None¶: If set, a list of fields that we should return distinct values of

limit: int¶: The maximum amount of entities that should be returned

startCursor: str | None = None¶: If set, we’ll only return entities that appear after this cursor in the index.

endCursor: str | None = None¶: If set, we’ll only return entities up to this cursor in the index.

currentCursor: str | None = None¶: Will be set after this query has been run, pointing after the last entity returned

__post_init__()¶

class core.db.SortOrder(*args, **kwds)¶

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

attribute access:
```
>>> Color.RED
<Color.RED: 1>
```
value lookup:
```
>>> Color(1)
<Color.RED: 1>
```
name lookup:
```
>>> Color['RED']
<Color.RED: 1>
```

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3

>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

Ascending = 1¶: Sort A->Z

Descending = 2¶: Sort Z->A

InvertedAscending = 3¶: Fetch Z->A, then flip the results (useful in pagination to go from a start cursor backwards)

InvertedDescending = 4¶: Fetch A->Z, then flip the results (useful in pagination)

core.db.acquire_transaction_success_marker()¶

Generates a token that will be written to the datastore (under “viur-transactionmarker”) if the transaction completes successfully. Currently only used by deferredTasks to check if the task should actually execute or if the transaction it was created in failed. :return: Name of the entry in viur-transactionmarker

Return type:: str

core.db.encodeKey(key)¶

Return the given key encoded as string (mimicking the old str() behaviour of keys)

Parameters:: key (core.db.types.Key)
Return type:: str

core.db.end_data_access_log(outer_access_log=None)¶

Retrieves the set of entries accessed so far.

To clean up and restart the log, call viur.datastore.startAccessDataLog().

If you called server.db.startAccessDataLog() before, you can re-apply the old log using the outerAccessLog param. Otherwise, it will disable the access log.

Parameters:

outerAccessLog – State of your log returned by server.db.startAccessDataLog()
outer_access_log (Optional[Set[Union[core.db.types.Key, str]]])

Returns:

t.Set of entries accessed

Return type:

Optional[Set[Union[core.db.types.Key, str]]]

core.db.endDataAccessLog(outerAccessLog=None)¶

Parameters:: outerAccessLog (Optional[Set[Union[core.db.types.Key, str]]])
Return type:: Optional[Set[Union[core.db.types.Key, str]]]

core.db.fix_unindexable_properties(entry, *, keep_exclusions=True)¶

Recursively walk the given Entity and add all properties to the list of unindexed properties if they contain a string longer than 1500 bytes (which is maximum size of a string that can be indexed). The datastore would return an error otherwise. https://cloud.google.com/datastore/docs/concepts/limits?hl=en#limits

Parameters:

entry (core.db.types.Entity) – The entity to fix (inplace)
keep_exclusions (bool) – If true, keep the properties already included in exclude_from_indexes. Otherwise, ignore them and exclude only non-indexable properties.

Returns:

The fixed entity

Return type:

core.db.types.Entity

core.db.get_or_insert(key, **kwargs)¶

Either creates a new entity with the given key, or returns the existing one.

Its guaranteed that there is no race-condition here; it will never overwrite a previously created entity. Extra keyword arguments passed to this function will be used to populate the entity if it has to be created; otherwise they are ignored.

Parameters:: key (core.db.types.Key) – The key which will be fetched or created.
Returns:: Returns the fetched or newly created Entity.
Return type:: core.db.types.Entity

core.db.GetOrInsert(key, **kwargs)¶

Parameters:

key (core.db.types.Key)
kwargs (Any)

Return type:

core.db.types.Entity

core.db.is_in_transaction()¶

Return type:: bool

core.db.IsInTransaction()¶

Return type:: bool

core.db.key_helper(in_key, target_kind, additional_allowed_kinds=(), adjust_kind=False)¶

Parameters:

in_key (Union[core.db.types.Key, str, int])
target_kind (str)
additional_allowed_kinds (Union[List[str], Tuple[str]])
adjust_kind (bool)

Return type:

core.db.types.Key

core.db.keyHelper(inKey, targetKind, additionalAllowedKinds=(), adjust_kind=False)¶

Parameters:

inKey (Union[core.db.types.Key, str, int])
targetKind (str)
additionalAllowedKinds (Union[List[str], Tuple[str]])
adjust_kind (bool)

Return type:

core.db.types.Key

core.db.normalize_key(key)¶

Normalizes a datastore key (replacing _application with the current one)

Parameters:: key (Union[None, core.db.types.Key, str]) – Key to be normalized.
Returns:: Normalized key in string representation.
Return type:: Union[None, core.db.types.Key]

core.db.normalizeKey(key)¶

Parameters:: key (Union[None, core.db.types.Key])
Return type:: Union[None, core.db.types.Key]

core.db.start_data_access_log()¶

Clears our internal access log (which keeps track of which entries have been accessed in the current request). The old set of accessed entries is returned so that it can be restored with server.db.popAccessData() in case of nested caching. You must call popAccessData afterwards, otherwise we’ll continue to log all entries accessed in subsequent request on the same thread! :return: t.Set of old accessed entries

Return type:: Set[Union[core.db.types.Key, str]]

core.db.startDataAccessLog()¶

Return type:: Set[Union[core.db.types.Key, str]]