BrightSign Asset Pool

the brightsign asset pool provides a reliable and resilient mechanism for downloading, storing and retrieving media assets and other files required by the player for its presentations core concepts the pool an asset pool is a directory hierarchy under a single root the directory is often simply named "pool", but the underlying implementation does not require that each asset is stored in one of a number of sub directories to improve performance on simple filesystems like fat the destination for a particular asset in the pool is determined by its hash https //en wikipedia org/wiki/cryptographic hash function (or digest) of its contents (or, in some cases, a hash of its metadata) which makes it a content addressable store in order to refer to an asset either before or after it has been downloaded it is necessary to know its hash (this is similar to the way that objects are stored in a git repository ) the pool can be configured with a maximum size and/or to ensure that a particular amount of space remains free on the storage device that contains it in order to stop the pool growing beyond these bounds old files are "pruned" files that must not be pruned must be "protected" before any operations are started that may write to the pool an asset that is stored in the pool using a particular hash will only be stored once until it is pruned requests to download an asset that is already in the pool will immediately succeed since the asset is already present this means that switching between presentations that share assets, even perhaps ones that haven't been used for a while, can avoid downloading assets unnecessarily asset collections an asset collection is a list of asset metadata the metadata either directly contains the hash as a specific field or the hash can be generated directly from other fields the metadata also provides a name for each asset which allows assets and therefore files in the pool to be looked up by name an asset collection is necessary to provide meaning for the assets stored in the pool asset collections themselves exist only in memory and are not stored in the pool itself synchronization specification synchronization specifications are often refered to as "sync specs" they consist of formatted xml or json that can be used to generate an asset collection but also have space for extra information that can be useful when downloading assets they are not recommended unless compatibility with existing systems that use them is required asset fetcher an asset fetcher runs asynchronously downloading any assets mentioned in an asset collection that are not currently available in the pool the file contents are automatically verified against the supplied hash during the download protocols assets are expected to primarily be downloaded using http or https however, the fetcher also supports ftp and file urls authentication arbitrary http headers can be provided when fetching all the assets in an asset collection the fetcher supports http basic, digest and ntlm authentication individual assets within a collection can be marked as opting out of providing these headers and authentication managing pool size the pool is intended to live for a long time with assets being fetched into it as required if the pool becomes too full then unused assets will be pruned to make space this means that any assets that are still required (or will be required) must be protected before performing any operations that may prune such as fetching or realizing the pool size can be restricted in two ways by requiring that a certain amount of space on the storage device always be kept free this means that space can be reserved for logs to grow into and any pool operations that can prune will ensure that at least the requested amount of space remains free checking the amount of free space on a storage device is a lightweight operation so this method should be preferred by setting a maximum size that the pool can occupy this can be used in combination with the previous method and attempting to exceed either threshold will trigger pruning calculating the size of the pool requires adding up the size of each file individually, so the first operation can take longer than usual once the first calculation is complete the pool size is only updated based on modifications to the pool so subsequent operations have less overhead files are considered for pruning based on how long ago they were last fetched, or would have been fetched had they not already been present in the pool the least recently fetched unprotected files will be pruned first note that unless the "working set" of assets is close to the maximum size of the pool this can often mean that any failures to protect assets have no ill effects referring to assets in the pool asset files can often be used directly from the pool the pool filename corresponding to an asset can be looked up by name in an asset collection sometimes it is necessary for the asset files to appear with their names within the filesystem asset files can be copied to a destination directory this is known as "realization" and should only be used when necessary since it can be very slow alternatively, an asset collection can be made to appear as a virtual filesystem hash uniqueness the hash for a particular asset file must be unique this happens automatically for normal hash algorithms such as sha 256 since hash collisions should not be possible if two files with different contents generate the same hash, this is known as a "hash collision" some older hash algorithms such as md5 and sha 1 are known to have hash collisions, but these only occur when files have been deliberately constructed to create such collisions – they do not occur by accident sometimes the actual hash of the asset is not available prior to the download in this case parts of the asset metadata, such as the url and a change hint are hashed together instead if the asset changes then some part of the metadata must also change so that the asset fetcher knows to download the asset again it is not possible to use http headers like etag or last modified as part of the hash since these aren't known prior to the download and can change independently asset metadata tag description name relative filename (may contain slashes but if so the file may not be realized ) asset names must be unique within an asset collection, but it is expected that different asset collections will contain assets with the same name the asset name is not stored in the pool, and the same asset file in the pool may be referenced by multiple names, even within the same asset collection size file length in bytes hash hash algorithm and hash of file contents link source url change hint an arbitrary value used by the client to ensure that files that have the same hash are not considered to be identical unless this value and the url also matches auth specify extra authentication information (see below) probe media probe data (useful for cheetah and panther) pattern wildcard pattern to match against hashing simple hash algorithms md5, sha1, sha224, sha256, sha384, sha512 besha1 algorithm this algorithm is only useful for referring to remote content that is too large to download during authoring its use is not recommended no hash if no hash is specified then the hash is generated internally by hashing together the "link" and "change hint" fields usage overview the brightsign asset pool can be used from brightscript or from javascript but it is not possible to share a single pool across both however, the brightscript rohtmlwidget mapfilefromassetpool method can be used to make a particular asset collection appear as files under a particular uri prefix brightscript tbd javascript create @brightsign/assetpool object to represent the asset pool passing the path to the pool directory create the asset collection data structure this is a javascript array of objects containing the metadata for the assets you wish to use protect the assets by calling the assetpool protectassets method to stop them being deleted in order to make space create a @brightsign/assetfetcher object passing the pool call the assetfetcher start method passing the asset collection any assets that are not present in the pool will start to be downloaded progress can be monitored by adding handlers for the progressevent and fileevent events if the fetch attempt completes successfully then the promise returned by assetfetcher start will be resolved if at least one of the assets could not be fully downloaded then the promise will be rejected when the promise returned by assetfetcher start resolves, the fetch attempt is complete create your set of assets as a javascript array of objects containing the metadata create a @brightsign/assetfiles object passing the pool and asset collection this object can be used to retrieve the full path to the files in the pool that correspond to the asset name specified see assetpoolfiles docid\ nhocamfrj0avahj vearr for more information create a @brightsign/assetrealizer object passing the pool and call the realize method if you wish to copy assets from the pool to a directory using the asset names note that this is slow and is only recommended for files that must exist in the filesystem such as brightsignos upgrade files (bsfw) create a @brightsign/assetcollectionview object passing the pool and the asset collection to create a virtual storage device containing the assets using the names and directory structure in the asset collection see assetpoolfiles docid\ nhocamfrj0avahj vearr for more information example brightsign asset pool example docid\ i3ekdhet5fkspb mvfjfl shows how the asset pool can be used from javascript it works from both nodejs and chromium, and can be run from the brightsign shell as follows node bs assetpool example js