:py:mod:`core.modules.file` =========================== .. py:module:: core.modules.file Module Contents --------------- Classes ~~~~~~~ .. autoapisummary:: core.modules.file.DownloadUrlBone core.modules.file.FileLeafSkel core.modules.file.FileNodeSkel core.modules.file.File Functions ~~~~~~~~~ .. autoapisummary:: core.modules.file.importBlobFromViur2 core.modules.file.thumbnailer core.modules.file.cloudfunction_thumbnailer core.modules.file.startCheckForUnreferencedBlobs core.modules.file.doCheckForUnreferencedBlobs core.modules.file.startCleanupDeletedFiles core.modules.file.doCleanupDeletedFiles core.modules.file.start_delete_pending_files core.modules.file.__getattr__ Attributes ~~~~~~~~~~ .. autoapisummary:: core.modules.file.VALID_FILENAME_REGEX core.modules.file.GOOGLE_STORAGE_CLIENT core.modules.file.PRIVATE_BUCKET_NAME core.modules.file.PUBLIC_BUCKET_NAME core.modules.file.PUBLIC_DLKEY_SUFFIX core.modules.file._private_bucket core.modules.file._public_bucket core.modules.file.FilePath .. py:data:: VALID_FILENAME_REGEX .. py:data:: GOOGLE_STORAGE_CLIENT .. py:data:: PRIVATE_BUCKET_NAME .. py:data:: PUBLIC_BUCKET_NAME .. py:data:: PUBLIC_DLKEY_SUFFIX :value: '_pub' .. py:data:: _private_bucket .. py:data:: _public_bucket .. py:data:: FilePath .. py:function:: importBlobFromViur2(dlKey, fileName) .. py:function:: thumbnailer(fileSkel, existingFiles, params) .. py:function:: cloudfunction_thumbnailer(fileSkel, existingFiles, params) External Thumbnailer for images. The corresponding cloudfunction can be found here . https://github.com/viur-framework/viur-cloudfunctions/tree/main/thumbnailer You can use it like so: main.py: .. code-block:: python from viur.core.modules.file import cloudfunction_thumbnailer conf.file_thumbnailer_url = "https://xxxxx.cloudfunctions.net/imagerenderer" conf.file_derivations = {"thumbnail": cloudfunction_thumbnailer} conf.derives_pdf = { "thumbnail": [{"width": 1920,"sites":"1,2"}] } skeletons/xxx.py: .. code-block:: python test = FileBone(derive=conf.derives_pdf) .. py:class:: DownloadUrlBone(*, compute = None, defaultValue = None, descr = None, getEmptyValueFunc = None, indexed = True, isEmptyFunc = None, languages = None, multiple = False, params = None, readOnly = None, required = False, searchable = False, type_suffix = '', unique = None, vfunc = None, visible = True, clone_behavior = None) Bases: :py:obj:`viur.core.bones.BaseBone` This bone is used to inject a freshly signed download url into a FileSkel. Initializes a new Bone. .. py:method:: unserialize(skel, name) Deserialize bone data from the datastore and populate the bone with the deserialized values. This function is the inverse of the serialize function. It converts data from the datastore into a format that can be used by the bones in the skeleton. :param skel: A SkeletonInstance object containing the values to be deserialized. :param name: The property name of the bone in its Skeleton (not the description). :returns: True if deserialization is successful, False otherwise. .. py:class:: FileLeafSkel(*args, **kwargs) Bases: :py:obj:`viur.core.prototypes.tree.TreeSkel` Default file leaf skeleton. .. py:attribute:: kindName :value: 'file' .. py:attribute:: size .. py:attribute:: dlkey .. py:attribute:: name .. py:attribute:: mimetype .. py:attribute:: weak .. py:attribute:: pending .. py:attribute:: width .. py:attribute:: height .. py:attribute:: downloadUrl .. py:attribute:: derived .. py:attribute:: pendingparententry .. py:attribute:: crc32c_checksum .. py:attribute:: md5_checksum .. py:attribute:: public .. py:attribute:: serving_url .. py:method:: _inject_serving_url(skel) :classmethod: Inject the serving url for public image files into a FileSkel .. py:method:: preProcessBlobLocks(locks) Ensure that our dlkey is locked even if we don't have a filebone here .. py:method:: refresh(skel) :classmethod: Refresh the bones current content. This function causes a refresh of all relational bones and their associated information. .. py:method:: write(skel, **kwargs) :classmethod: Write current Skeleton to the datastore. Stores the current data of this instance into the database. If an *key* value is set to the object, this entity will ne updated; Otherwise a new entity will be created. To read a Skeleton object from the data store, see :func:`~viur.core.skeleton.Skeleton.read`. :param key: Allows to specify a key that is set to the skeleton and used for writing. :param update_relations: If False, this entity won't be marked dirty; This avoids from being fetched by the background task updating relations. :returns: The Skeleton. .. py:class:: FileNodeSkel(*args, **kwargs) Bases: :py:obj:`viur.core.prototypes.tree.TreeSkel` Default file node skeleton. .. py:attribute:: kindName :value: 'file_rootNode' .. py:attribute:: name .. py:attribute:: rootNode .. py:attribute:: public .. py:attribute:: viurCurrentSeoKeys .. py:class:: File(moduleName, modulePath, *args, **kwargs) Bases: :py:obj:`viur.core.prototypes.tree.Tree` Tree module prototype. It is used for hierarchical structures, either as a tree with nodes and leafs, or as a hierarchy with nodes only. .. py:attribute:: PENDING_POSTFIX :value: ' (pending)' .. py:attribute:: DOWNLOAD_URL_PREFIX :value: '/file/download/' .. py:attribute:: INTERNAL_SERVING_URL_PREFIX :value: '/file/serve/' .. py:attribute:: MAX_FILENAME_LEN :value: 256 .. py:attribute:: IMAGE_META_MAX_SIZE :type: viur.core.decorators.t.Final[int] Maximum size of image files that should be analysed in :meth:`set_image_meta`. Default: 10 MiB .. py:attribute:: leafSkelCls .. py:attribute:: nodeSkelCls .. py:attribute:: handler :value: 'tree.simple.file' .. py:attribute:: adminInfo .. py:attribute:: roles .. py:attribute:: default_order :value: 'name' .. py:attribute:: SERVE_VALID_OPTIONS Valid modification option shorts for the serve-function. This is passed-through to the Google UserContent API, and hast to be supported there. .. py:attribute:: SERVE_VALID_FORMATS Valid file-formats to the serve-function. This is passed-through to the Google UserContent API, and hast to be supported there. .. py:method:: get_bucket(dlkey) :staticmethod: Retrieves a Google Cloud Storage bucket for the given dlkey. .. py:method:: is_valid_filename(filename) :staticmethod: Verifies a valid filename. The filename should be valid on Linux, Mac OS and Windows. It should not be longer than MAX_FILENAME_LEN chars. Rule set: https://stackoverflow.com/a/31976060/3749896 Regex test: https://regex101.com/r/iBYpoC/1 .. py:method:: hmac_sign(data) :staticmethod: .. py:method:: hmac_verify(data, signature) :staticmethod: .. py:method:: create_internal_serving_url(serving_url, size = 0, filename = '', options = '', download = False) :staticmethod: Helper function to generate an internal serving url (endpoint: /file/serve) from a Google serving url. This is needed to hide requests to Google as they are internally be routed, and can be the result of a legal requirement like GDPR. :param serving_url: Is the original serving URL as generated from FileLeafSkel._inject_serving_url() :param size: Optional size setting :param filename: Optonal filename setting :param options: Additional options parameter-pass through to /file/serve :param download: Download parameter-pass through to /file/serve .. py:method:: create_download_url(dlkey, filename, derived = False, expires = datetime.timedelta(hours=1), download_filename = None) :staticmethod: Utility function that creates a signed download-url for the given folder/filename combination :param folder: The GCS-Folder (= the download-key) for that file :param filename: The name of the file. Either the original filename or the name of a derived file. :param derived: True, if it points to a derived file, False if it points to the original uploaded file :param expires: None if the file is supposed to be public (which causes it to be cached on the google ede caches), otherwise a datetime.timedelta of how long that link should be valid :param download_filename: If set, browser is enforced to download this blob with the given alternate filename :return: The signed download-url relative to the current domain (eg /download/...) .. py:method:: parse_download_url(url) :staticmethod: Parses a file download URL in the format `/file/download/xxxx?sig=yyyy` into its FilePath. If the URL cannot be parsed, the function returns None. :param url: The file download URL to be parsed. :return: A FilePath on success, None otherwise. .. py:method:: create_src_set(file, expires = datetime.timedelta(hours=1), width = None, height = None, language = None) :staticmethod: Generates a string suitable for use as the srcset tag in html. This functionality provides the browser with a list of images in different sizes and allows it to choose the smallest file that will fill it's viewport without upscaling. :param file: The file skeleton (or if multiple=True a single value from it) to generate the srcset. :param expires: None if the file is supposed to be public (which causes it to be cached on the google edecaches), otherwise it's lifetime in seconds :param width: A list of widths that should be included in the srcset. If a given width is not available, it will be skipped. :param height: A list of heights that should be included in the srcset. If a given height is not available, it will be skipped. :param language: Language overwrite if file has multiple languages, and we want to explicitly specify one :return: The srctag generated or an empty string if a invalid file object was supplied .. py:method:: write(filename, content, mimetype = 'text/plain', width = None, height = None, public = False) Write a file from any buffer into the file module. :param filename: Filename to be written. :param content: The file content to be written, as bytes-like object. :param mimetype: The file's mimetype. :param width: Optional width information for the file. :param height: Optional height information for the file. :param public: True if the file should be publicly accessible. :return: Returns the key of the file object written. This can be associated e.g. with a FileBone. .. py:method:: read(key = None, path = None) Read a file from the Cloud Storage. If a key and a path are provided, the key is preferred. This means that the entry in the db is searched first and if this is not found, the path is used. :param key: Key of the LeafSkel that contains the "dlkey" and the "name". :param path: The path of the file in the Cloud Storage Bucket. :return: Returns the file as a io.BytesIO buffer and the content-type .. py:method:: deleteRecursive(parentKey) Recursively processes a delete request. This will delete all entries which are children of *nodeKey*, except *key* nodeKey. :param parentKey: URL-safe key of the node which children should be deleted. .. py:method:: getUploadURL(fileName, mimeType, size = None, node = None, authData = None, authSig = None, public = False) .. py:method:: download(blobKey, fileName = '', download = False, sig = '', *args, **kwargs) Download a file. :param blobKey: The unique blob key of the file. :param fileName: Optional filename to provide in the header. :param download: Set header to attachment retrival, set explictly to "1" if download is wanted. .. py:method:: serve(host, key, size = None, filename = None, options = '', download = False) Requests an image using the serving url to bypass direct Google requests. :param host: the google host prefix i.e. lh3 :param key: the serving url key :param size: the target image size :param filename: a random string with an extention, valid extentions are (defined in File.SERVE_VALID_FORMATS). :param options: - seperated options (defined in File.SERVE_VALID_OPTIONS). c - crop p - face crop fv - vertrical flip fh - horizontal flip rXXX - rotate 90, 180, 270 nu - no upscale :param download: Serves the content as download (Content-Disposition) or not. :return: Returns the requested content on success, raises a proper HTTP exception otherwise. .. py:method:: add(skelType, node = None, *args, **kwargs) Add a new entry with the given parent *node*, and render the entry, eventually with error notes on incorrect data. Data is taken by any other arguments in *kwargs*. The function performs several access control checks on the requested entity before it is added. .. seealso:: :func:`canAdd`, :func:`onAdd`, , :func:`onAdded` :param skelType: Defines the type of the new entry and may either be "node" or "leaf". :param node: URL-safe key of the parent. :returns: The rendered, added object of the entry, eventually with error hints. :raises: :exc:`viur.core.errors.NotAcceptable`, when no valid *skelType* was provided. :raises: :exc:`viur.core.errors.NotFound`, when no valid *node* was found. :raises: :exc:`viur.core.errors.Unauthorized`, if the current user does not have the required permissions. :raises: :exc:`viur.core.errors.PreconditionFailed`, if the *skey* could not be verified. .. py:method:: get_download_url(key = None, dlkey = None, filename = None, derived = False) Request a download url for a given file :param key: The key of the file :param dlkey: The download key of the file :param filename: The filename to be given. If no filename is provided downloadUrls for all derived files are returned in case of `derived=True`. :param derived: True, if a derived file download URL is being requested. .. py:method:: onEdit(skelType, skel) Hook function that is called before editing an entry. It can be overridden for a module-specific behavior. :param skelType: Defines the type of the node that shall be edited. :param skel: The Skeleton that is going to be edited. .. seealso:: :func:`edit`, :func:`onEdited` .. py:method:: onAdded(skelType, skel) Hook function that is called after adding an entry. It should be overridden for a module-specific behavior. The default is writing a log entry. :param skelType: Defines the type of the node that has been added. :param skel: The Skeleton that has been added. .. seealso:: :func:`add`, :func:`onAdd` .. py:method:: set_image_meta(key) Write image metadata (height and width) to FileSkel .. py:method:: mark_for_deletion(dlkey) Adds a marker to the datastore that the file specified as *dlkey* can be deleted. Once the mark has been set, the data store is checked four times (default: every 4 hours) if the file is in use somewhere. If it is still in use, the mark goes away, otherwise the mark and the file are removed from the datastore. These delayed checks are necessary due to database inconsistency. :param dlkey: Unique download-key of the file that shall be marked for deletion. .. py:function:: startCheckForUnreferencedBlobs() Start searching for blob locks that have been recently freed .. py:function:: doCheckForUnreferencedBlobs(cursor=None) .. py:function:: startCleanupDeletedFiles() Increase deletion counter on each blob currently not referenced and delete it if that counter reaches maxIterCount .. py:function:: doCleanupDeletedFiles(cursor=None) .. py:function:: start_delete_pending_files() Start deletion of pending FileSkels that are older than 7 days. .. py:function:: __getattr__(attr)