node scalable blob store
Last release prior to upgrading to async
A file system blob store that is designed to prevent conflicts when used with a distributed file system or storage area network.
Please Star on GitHub / NPM and Watch for updates.
Note: Requires Node.js v12 or later.
npm install scalable-blob-store --save
const os = require('os');
const ulid = require('ulid').ulid; // You need a unique ID generator function
const BlobStore = require('scalable-blob-store');
const options = {
blobStoreRoot: os.tmpdir() + '/blobs', // Change this!
idFunction: ulid,
dirDepth: 4,
dirWidth: 1000,
};
// Creating the blobStore Object
const blobStore = new BlobStore(options);
const result = await blobStore.createWriteStream();
console.dir(result);
// Logs the result object which contains the blobPath and writeStream.
// Use the writeStream to save your blob.
// Store the blobPath in your database.
//
// result object will be similar to this:
// {
// blobPath: "/01CTZRTWMAD153V20K26S4Y0BW/01CTZRTWMBZW4SPR4E5QGGJYSH/01CTZRTWMB3QXZK04SYFY8ZJVR/01CTZS3KJYFPRQ34S3T15Y798S",
// writeStream: [WriteStream]
// }
//
// In this example the full file path for the blob would be something like this:
// /tmp/blobs/01CTZRTWMAD153V20K26S4Y0BW/01CTZRTWMBZW4SPR4E5QGGJYSH/01CTZRTWMB3QXZK04SYFY8ZJVR/01CTZS3KJYFPRQ34S3T15Y798S
//
// This is based on the blobStoreRoot + blobPath.See the Quick Start example files for more detail:
After researching user file storage, or blob storage, for a web application I was working on I discovered the most common solution used by web developers is to store files using a cloud service provider. After creating an account with such providers as Amazon S3, Google Cloud Storage, or Azure Storage, they just stash all their application files and blobs there.
I researched the price of cloud storage and decided I wanted a free local version that would scale if needed.
I looked at a number of existing solutions such as filestorage but was unhappy with the scalability of these solutions. Most are only designed for a single server and would cause write conflicts if a distributed file system, cluster file system like GlusterFS, or a storage area network was used as the backend file system.
On a long car trip I was thinking about a solution for my blob storage and came up with scalable-blob-store.
To achieve scalability on a distributed or replicated file system, scalable-blob-store does not use index files or other databases to manage the files on the disk or storage system. Instead, the file system itself is used to find the latest storage path based on the file systems birthtime attribute (the directory creation date).
Once the latest path has been determined, the number of files within the directory are counted to ensure it remains under the configured value. This is to prevent disk performance issues when very large numbers of files are stored within a single directory. If the number of items within a directory becomes too large, a new storage path is determined.
Because there are no databases used to manage the files in the root path, it is up to you to maintain the returned blobPath value and metadata about the stored files in your own database.
The reason scalable-blob-store is scalable is due to the naming of the directories and files within your file system. Every directory and file saved to disk is named by a generated unique id based on a user defined funciton. You could use any unique id generator such as ULID, CUID, UUID v4, or MongoDBs ObjectIds just to name a few. Check out my Awesome Unique ID repository for more examples. Merging directories between servers or disks should never cause file name collisions.
If a replicated or cluster file system is in use the only conflict that can occur is when one server is reading a file while another is removing the same file. scalable-blob-store does not try to manage this conflict, however it will raise the exception.
Below are examples of the directory structure created by scalable-blob-store.
Example with CUID directory and file names:
blobscij50xia200pzzph3we9r62bi // ← Directory File ↓
blobscij50xia300q1zph3m4df4ypz..cij50xiae00qgzph3i0ms0l2wExample with UUID directory and file names:
blobs846a291f-9864-40bb-aefe-f29bdc73a761 // ← Directory File ↓
blobs846a291f-9864-40bb-aefe-f29bdc73a761..8b86b6fe-6166-424c-aed9-8faf1e62689escalable-blob-store supports configuration options to give you control over the directory and file ids used, depth of the directory structure, and the width of the directories. The default options give 3 directories deep containing 1000 items giving a total storage of one billion files within the directory structure.
Other operational points of interest:
dirWidth value, the next directory is created.dirWidth value, the next parent directory is created.dirWidth value, the dirWidth value is ignored.Write
On my laptop with an M.2 SSD disk, running the test-fs.js script produces the following results:
====================================================================================================
Testing scalable-blob-store with the following options:
blobStoreRoot: /tmp/blobs/test-fs
idFunction: ulid
dirDepth: 3
dirWidth: 1000
repeat: 10000
Beginning test...
====================================================================================================
Test complete.
====================================================================================================
{
blobStoreRoot: '/tmp/blobs/test-fs',
dirDepth: 3,
dirWidth: 1000,
runTimeMilliseconds: 83730,
totalDirectories: 12,
totalFiles: 10000,
totalBytes: 430000,
lastBlobPath: '/ckxwcwgwz0001lk9hgq8t9iup/ckxwcwgx00002lk9h6tbpdmq1/ckxwcy36m06yclk9hb0g92dwg/ckxwcy9ip07q4lk9h5uyl10k6'
}
====================================================================================================
Please remove /tmp/blobs/test-fs manually.
====================================================================================================
Read
Read performance will be close to, if not the same, as disk speed.
All the BlobStore methods within scalable-blob-store return a Promise. This is perfect for using with the async/await language features.
| API | Type | Returns |
|---|---|---|
| new BlobStore(options) | Constructor | blobStore Instance |
| blobStore.blobStoreRoot | Read Only Property | String |
| blobStore.idFunction | Read Only Property | Function |
| blobStore.dirDepth | Read Only Property | Number |
| blobStore.dirWidth | Read Only Property | Number |
| blobStore.getCurrentBlobDir() | Method | Promise<String> |
| blobStore.setCurrentBlobDir(blobDir) | Method | Promise<undefined> |
| blobStore.createWriteStream() | Method | Promise<Object> |
| blobStore.write(data, writeOptions) | Method | Promise<String> |
| blobStore.append(blobPath, data, appendOptions) | Method | Promise<undefined> |
| blobStore.copy(blobPath, flags) | Method | Promise<String> |
| blobStore.createReadStream(blobPath) | Method | Promise<ReadStream> |
| blobStore.read(blobPath, readOptions) | Method | Promise<data> |
| blobStore.open(blobPath, flags, mode) | Method | Promise<FileHandle> |
| blobStore.realPath(blobPath, realPathOptions) | Method | Promise<String> |
| blobStore.stat(blobPath) | Method | Promise<Stats> |
| blobStore.exists(blobPath) | Method | Promise<Boolean> |
| blobStore.remove(blobPath) | Method | Promise<undefined> |
new BlobStore(options)Type: Constructor function.
Parameter: options as an Object.
Returns: A new BlobStore object to be used to store data.
Description:
You can call new BlobStore(options) multiple times to create more than one blob store.
Options are passed to the constructor function as a JavaScript object.
| Key | Description | Defaults |
|---|---|---|
blobStoreRoot |
Root directory to store blobs | Required |
idFunction |
Any ID function that returns a unique ID string | Required |
dirDepth |
How deep you want the directories under the root | 3 |
dirWidth |
The maximum number of files or directories in a directory | 1000 |
Example:
// Start by requiring the `scalable-blob-store` constructor function:
const BlobStore = require('scalable-blob-store');
// You will need a unique ID function
const uuid = require('uuid');
// Create the options object
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 4,
dirWidth: 2000,
};
// Create a blob store using the options `object`:
const blobStore = new BlobStore(options);Creating multiple blob stores:
const userOptions = {
blobStoreRoot: '/app/blobs/user',
idFunction: uuid.v4,
dirDepth: 4,
dirWidth: 2000,
};
const pdfOptions = {
blobStoreRoot: '/app/blobs/pdf',
idFunction: uuid.v4,
dirDepth: 2,
dirWidth: 300,
};
const userFileStore = new BlobStore(userOptions);
const pdfDocumentStore = new BlobStore(pdfOptions);blobStoreRootType: Read only property.
Returns: A String that matches your options.blobStoreRoot value.
Description:
This is a convenience property to allow you to pass the blobStore object to a sub module and still have access to the configured properties.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 4,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
console.log(blobStore.blobStoreRoot);
// Outputs '/app/blobs' which you configured in the optionsidFunctionType: Read only property.
Returns: The unique ID function you configured in the options.idFunction value.
Description:
This is a convenience property to allow you to pass the blobStore object to a sub module and still have access to the configured properties.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 4,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
console.log(blobStore.idFunction());
// Outputs 'bac00ab2-5e6d-4b77-bfa4-e9befc3e4279' which is a generated UUID from the idFunction.dirDepthType: Read only property.
Returns: A Number that matches your options.dirDepth value.
Description:
This is a convenience property to allow you to pass the blobStore object to a sub module and still have access to the configured properties.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 4,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
console.log(blobStore.dirDepth);
// Outputs '4' which you configured in the optionsdirWidthType: Read only property.
Returns: A Number that matches your options.dirWidth value.
Description:
This is a convenience property to allow you to pass the blobStore object to a sub module and still have access to the configured properties.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 4,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
console.log(blobStore.dirWidth);
// Outputs '2000' which you configured in the optionsgetCurrentBlobDir()Type: Method.
Returns: A Promise that resolves to a String that is the current active blob creation directory.
Description:
This function is used internally by the BlobStore to determine the directory where the next blob file will be saved to disk.
If you ever need to store a blob file outside of the BlobStore you could use this method to locate the right place to put your file.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
try {
console.log(await blobStore.getCurrentBlobDir());
// The 'dirDepth' option above is set to 3 so the output will be similar to the following:
// '/e44d3b0d-b552-4257-8b64-a53331184c38/443061b9-bfa7-40fc-a5a9-d848bc52155e/4d818f4c-88b3-45fd-a104-a2fc3700e9de'
} catch (err) {
console.error(err);
}
}
main();setCurrentBlobDir(blobDir)Type: Method.
Parameters: blobDir as a String.
blobStoreRoot path.Returns: A Promise that resolves to undefined.
Description:
This function can be used to guide the BlobStore to save new blob files into a desired blobPath.
One issue with scalable-blob-store is that if you remove many blob files the directories the files were located in will not be removed.
You could either remove the directories yourself, or repopulate them with new blob files by setting the current active blob directory.
This function was added to enable consumers of this module to work around empty blob directories.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
try {
console.log(await blobStore.getCurrentBlobDir());
// The 'dirDepth' option above is set to 3 so the output will be similar to the following:
// '/e44d3b0d-b552-4257-8b64-a53331184c38/443061b9-bfa7-40fc-a5a9-d848bc52155e/4d818f4c-88b3-45fd-a104-a2fc3700e9de'
await blobStore.setCurrentBlobDir('/some/blob/path');
console.log(await blobStore.getCurrentBlobDir());
// Outputs '/some/blob/path' to the console.
// Any new blob files added to the blob store will go into this path until there are `dirWidth` or 2000 files within it.
} catch (err) {
console.error(err);
}
}
main();createWriteStream()Type: Method.
Returns: A Promise that resolves to an Object containing the child path to the file within the blob store root and a WriteStream.
Description:
Here is an exampe of the returned object using UUID as the idFunction:
{
blobPath: "/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19",
writeStream: stream.Writable
}Use the writeStream to save your blob or file.
The blobPath needs to be saved to your database for future access.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
// The below readStream is simply to make this a complete example
const fs = require('fs');
const readStream = fs.createReadStream('/path/to/file');
async function main() {
let result;
try {
result = await blobStore.createWriteStream();
} catch (err) {
console.error(err);
}
console.dir(result);
// result object will be similar to this:
// {
// blobPath: "/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19",
// writeStream: [WriteStream]
// }
// Using a Promise to encapsulate the write asynchronous events.
await new Promise((resolve, reject) => {
result.writeStream.on('finish', () => {
resolve();
});
result.writeStream.on('error', reject);
readStream.pipe(result.writeStream);
});
console.log(blobPath);
// Logs the blobPath. Save this in your database.
}
main();write(data, writeOptions)Type: Method.
Parameter: data as either String, Buffer, TypedArray, or DataView.
Parameter: writeOptions as an Object.
writeOptions object supports an encoding, mode, and flag property.Returns: A Promise that resolves to a String.
blobPath value which needs committing to your database.Description:
If you have simple data in memory rather than a stream of data you can use this method to store the data into a blob file.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
const data = 'The quick brown fox jumps over the lazy dog.';
try {
const blobPath = await blobStore.write(data);
// The returned blobPath will look something like this:
// '/e44d3b0d-b552-4257-8b64-a53331184c38/443061b9-bfa7-40fc-a5a9-d848bc52155e/4d818f4c-88b3-45fd-a104-a2fc3700e9de'
// Save it to your database.
} catch (err) {
console.error(err);
}
}
main();append(blobPath, data, appendOptions)Type: Method.
Parameter: blobPath as a String.
blobPath from your application database.Parameter: data as either a String or Buffer.
Parameter: appendOptions as an Object.
appendOptions object supports an encoding, mode, and flag property.Returns: A Promise that resolves to a undefined.
Description:
Use this method to add simple in memory data to the end of the blob file.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
const data = 'The quick brown fox jumps over the lazy dog.';
try {
await blobStore.append(data);
} catch (err) {
console.error(err);
}
}
main();copy(blobPath, flags)Type: Method.
Parameter: blobPath as a String.
blobPath from your application database.Parameter: flags as a Number.
Returns: A Promise that resolves to a String.
blobPath value for the copied blob file.Description:
Use this method to create a copy of an existing blob file.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
try {
const blobPathSource =
'/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';
const blobPathDest = await blobStore.copy(blobPathSource);
// Store your new blobPath into your application database
} catch (err) {
console.error(err);
}
}
main();createReadStream(blobPath)Type: Method.
Parameter: blobPath as a String.
blobPath from your application database.Returns: A Promise that resolves to a ReadStream.
Description:
Creates a readable stream to the blob file located at the blobPath.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
async function main() {
// Get the blobPath value from your database.
const blobPath =
'/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19h';
let readStream;
try {
readStream = await blobStore.createReadStream(blobPath);
} catch (err) {
console.error(err);
}
readStream.on('error', (err) => {
console.error(err);
});
// Blob contents is piped to the console.
readStream.pipe(process.stdout);
}
main();read(blobPath, readOptions)Type: Method.
Parameter: blobPath as a String.
blobPath from your application database.Parameter: readOptions as an Object.
Returns: A Promise that resolves to a the contents of the blob file.
scalable-blob-store sets the readOptions.encoding value to 'utf8' by default.Description:
Use this method to read the content of a small blob file into memory.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
try {
// Retrieve the blobPath value from your database
const blobPath =
'/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';
const content = await blobStore.read(blobPath);
// Do something with the content
} catch (err) {
console.error(err);
}
}
main();open(blobPath, flags, mode)Type: Method.
Parameter: blobPath as a String.
blobPath from your application database.Parameter: flags as an String or Number.
Returns: A Promise that resolves to a FileHandle object.
Description:
This is a more advanced method allowing you to carry out many file operations against the blob file.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
try {
// Retrieve the blobPath value from your database
const blobPath =
'/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';
const fileHandle = await blobStore.open(blobPath);
// Do something with the file handle object
// See the documentation for more detail
// The documentation link is in the description above
} catch (err) {
console.error(err);
}
}
main();realPath(blobPath, realPathOptions)Type: Method.
Parameter: blobPath as a String.
blobPath from your application database.Parameter: realPathOptions as a String or Object.
Returns: A Promise that resolves to a String.
Description:
Use this method to locate a blob file on the file system. This method should not really be needed because you can determine the full blob file path. Simply concatenate the blobStoreRoot and the blobPath values.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
try {
// Retrieve the blobPath value from your database
const blobPath =
'/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';
const fsPath = await blobStore.realPath(blobPath);
// With the above options the result will be similar to this:
// '/app/blobs/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19
} catch (err) {
console.error(err);
}
}
main();stat(blobPath)Type: Method.
Parameter: blobPath as a String.
Returns: A stats Object.
Description:
Rather than parse the file system stats object, scalable-blob-store returns the raw stats object.
More stat class details can be found on Wikipedia.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
try {
// Retrieve the blobPath value from your database
const blobPath =
'/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';
const stats = await blobStore.stat(blobPath);
console.dir(stats);
// Console output will be similar to the following.
// { dev: 2050,
// mode: 33188,
// nlink: 1,
// uid: 1000,
// gid: 1000,
// rdev: 0,
// blksize: 4096,
// ino: 6707277,
// size: 44,
// blocks: 8,
// atime: Mon Oct 12 2015 08:51:29 GMT+1000 (AEST),
// mtime: Mon Oct 12 2015 08:51:29 GMT+1000 (AEST),
// ctime: Mon Oct 12 2015 08:51:29 GMT+1000 (AEST),
// birthtime: Mon Oct 12 2015 08:51:29 GMT+1000 (AEST) }
} catch (err) {
console.error(err);
}
}
main();exists(blobPath)Type: Method.
Parameter: blobPath as a String.
Returns: Boolean
true if the file exists, otherwise false.Description:
Use this method for a simple blob file existence test.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
try {
// Retrieve the blobPath value from your database
const blobPath =
'/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';
const exists = await blobStore.exists(blobPath);
// The result will be either true or false depending if the blob file exists.
} catch (err) {
console.error(err);
}
}
main();remove(blobPath)Type: Method.
Parameter: blobPath as a String.
Returns: undefined if nothing went wrong or the file did not exist.
Description:
Use this method to delete a blob file. This method can not be used to remove directories.
Example:
const BlobStore = require('scalable-blob-store');
const uuid = require('uuid');
const options = {
blobStoreRoot: '/app/blobs',
idFunction: uuid.v4,
dirDepth: 3,
dirWidth: 2000,
};
const blobStore = new BlobStore(options);
async function main() {
try {
// Retrieve the blobPath value from your database
const blobPath =
'/e6b7815a-c818-465d-8511-5a53c8276b86/aea4be6a-9e7f-4511-b394-049e68f59b02/fea722d1-001a-4765-8408-eb8e0fe7dbc6/183a6b7b-2fd6-4f80-8c6a-2647beb7bb19';
await blobStore.remove(blobPath);
// The blob file will no longer exist
} catch (err) {
console.error(err);
}
}
main();There is a minor issue in scalable-blob-store. If there are a large number of blob files added and then removed from the blob store, you may have empty directories or directories with a small number of files in them. These directories will never be removed and will not be populated.
If you wish to prevent empty or sparsely populated directories you will need to run a maintenance task against the blobStoreRoot directory. This maintenance task will need to look for empty or incomplete directories and call the setCurrentBlobDir method passing in the empty blobPath.
For your application you may find you rarely remove large numbers of blob files. If this is the case then this issue can be ignored.
There are two methods for testing scalable-blob-store:
os.tmpdir() directory.After cloning scalable-blob-store, type the following into your console:
npm install
npm test
Running the test-fs.js file will create a ~/blobs directory in your temporary directory and then recursively fill it with lots of blobs.
The default options configured in the test-fs.js file are:
const opts = {
blobStoreRoot: os.tmpdir() + '/blobs',
idFunction: cuid,
dirDepth: 3,
dirWidth: 1000,
};
const repeat = 10000;Change the options if you wish to see different results.
After cloning scalable-blob-store, type the following into your console:
npm install
node ./tests/test-fs.js
Once complete, inspect the /tmp/blobs directory. I suggest using the tree command which gives you a summary of directories and files within the target directory.
tree ~/blobs
tree -d ~/blobs
I, Grant Carthew, am a technologist from Queensland, Australia. I work on code in a number of personal projects and when the need arises I build my own packages.
This project exists because I needed a local blob store that could scale.
Everything I do in open source is done in my own time and as a contribution to the open source community.
If you are using my projects and would like to thank me or support me, please click the Patreon link below.
See my other projects on NPM.
git checkout -b my-new-featuregit commit -am 'Add some feature'git push origin my-new-featurenode-uuid with uuid.ES5 build code. Removed Nodejs engine requirements.return null statements.return null after resolve/reject calls to prevent Bluebird warnings.es5dist for older versions of node. Packages updated.MIT
