Skip to content
This repository has been archived by the owner on Aug 11, 2020. It is now read-only.

Commit

Permalink
fs: introduce opendir() and fs.Dir
Browse files Browse the repository at this point in the history
This adds long-requested methods for asynchronously interacting and
iterating through directory entries by using `uv_fs_opendir`,
`uv_fs_readdir`, and `uv_fs_closedir`.

`fs.opendir()` and friends return an `fs.Dir`, which contains methods
for doing reads and cleanup. `fs.Dir` also has the async iterator
symbol exposed.

The `read()` method and friends only return `fs.Dirent`s for this API.
Having a entry type or doing a `stat` call is deemed to be necessary in
the majority of cases, so just returning dirents seems like the logical
choice for a new api.

Reading when there are no more entries returns `null` instead of a
dirent. However the async iterator hides that (and does automatic
cleanup).

The code lives in separate files from the rest of fs, this is done
partially to prevent over-pollution of those (already very large)
files, but also in the case of js allows loading into `fsPromises`.

Due to async_hooks, this introduces a new handle type of `DIRHANDLE`.

This PR does not attempt to make complete optimization of
this feature. Notable future improvements include:
- Moving promise work into C++ land like FileHandle.
- Possibly adding `readv()` to do multi-entry directory reads.
- Aliasing `fs.readdir` to `fs.scandir` and doing a deprecation.

Refs: nodejs/node-v0.x-archive#388
Refs: nodejs/node#583
Refs: libuv/libuv#2057

PR-URL: nodejs/node#29349
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: David Carlier <devnexen@gmail.com>
  • Loading branch information
Fishrock123 authored and addaleax committed Oct 9, 2019
1 parent 968a23c commit 91561d0
Show file tree
Hide file tree
Showing 19 changed files with 1,165 additions and 119 deletions.
6 changes: 6 additions & 0 deletions doc/api/errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -834,6 +834,11 @@ A signing `key` was not provided to the [`sign.sign()`][] method.
[`crypto.timingSafeEqual()`][] was called with `Buffer`, `TypedArray`, or
`DataView` arguments of different lengths.

<a id="ERR_DIR_CLOSED"></a>
### ERR_DIR_CLOSED

The [`fs.Dir`][] was previously closed.

<a id="ERR_DNS_SET_SERVERS_FAILED"></a>
### ERR_DNS_SET_SERVERS_FAILED

Expand Down Expand Up @@ -2454,6 +2459,7 @@ such as `process.stdout.on('data')`.
[`dgram.disconnect()`]: dgram.html#dgram_socket_disconnect
[`dgram.remoteAddress()`]: dgram.html#dgram_socket_remoteaddress
[`errno`(3) man page]: http://man7.org/linux/man-pages/man3/errno.3.html
[`fs.Dir`]: fs.html#fs_class_fs_dir
[`fs.readFileSync`]: fs.html#fs_fs_readfilesync_path_options
[`fs.readdir`]: fs.html#fs_fs_readdir_path_options_callback
[`fs.symlink()`]: fs.html#fs_fs_symlink_target_path_type_callback
Expand Down
216 changes: 214 additions & 2 deletions doc/api/fs.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,13 +284,148 @@ synchronous use libuv's threadpool, which can have surprising and negative
performance implications for some applications. See the
[`UV_THREADPOOL_SIZE`][] documentation for more information.

## Class fs.Dir
<!-- YAML
added: REPLACEME
-->

A class representing a directory stream.

Created by [`fs.opendir()`][], [`fs.opendirSync()`][], or [`fsPromises.opendir()`][].

Example using async interation:

```js
const fs = require('fs');

async function print(path) {
const dir = await fs.promises.opendir(path);
for await (const dirent of dir) {
console.log(dirent.name);
}
}
print('./').catch(console.error);
```

### dir.path
<!-- YAML
added: REPLACEME
-->

* {string}

The read-only path of this directory as was provided to [`fs.opendir()`][],
[`fs.opendirSync()`][], or [`fsPromises.opendir()`][].

### dir.close()
<!-- YAML
added: REPLACEME
-->

* Returns: {Promise}

Asynchronously close the directory's underlying resource handle.
Subsequent reads will result in errors.

A `Promise` is returned that will be resolved after the resource has been
closed.

### dir.close(callback)
<!-- YAML
added: REPLACEME
-->

* `callback` {Function}
* `err` {Error}

Asynchronously close the directory's underlying resource handle.
Subsequent reads will result in errors.

The `callback` will be called after the resource handle has been closed.

### dir.closeSync()
<!-- YAML
added: REPLACEME
-->

Synchronously close the directory's underlying resource handle.
Subsequent reads will result in errors.

### dir.read([options])
<!-- YAML
added: REPLACEME
-->

* `options` {Object}
* `encoding` {string|null} **Default:** `'utf8'`
* Returns: {Promise} containing {fs.Dirent}

Asynchronously read the next directory entry via readdir(3) as an
[`fs.Dirent`][].

A `Promise` is returned that will be resolved with a [Dirent][] after the read
is completed.

_Directory entries returned by this function are in no particular order as
provided by the operating system's underlying directory mechanisms._

### dir.read([options, ]callback)
<!-- YAML
added: REPLACEME
-->

* `options` {Object}
* `encoding` {string|null} **Default:** `'utf8'`
* `callback` {Function}
* `err` {Error}
* `dirent` {fs.Dirent}

Asynchronously read the next directory entry via readdir(3) as an
[`fs.Dirent`][].

The `callback` will be called with a [Dirent][] after the read is completed.

The `encoding` option sets the encoding of the `name` in the `dirent`.

_Directory entries returned by this function are in no particular order as
provided by the operating system's underlying directory mechanisms._

### dir.readSync([options])
<!-- YAML
added: REPLACEME
-->

* `options` {Object}
* `encoding` {string|null} **Default:** `'utf8'`
* Returns: {fs.Dirent}

Synchronously read the next directory entry via readdir(3) as an
[`fs.Dirent`][].

The `encoding` option sets the encoding of the `name` in the `dirent`.

_Directory entries returned by this function are in no particular order as
provided by the operating system's underlying directory mechanisms._

### dir\[Symbol.asyncIterator\]()
<!-- YAML
added: REPLACEME
-->

* Returns: {AsyncIterator} to fully iterate over all entries in the directory.

_Directory entries returned by this iterator are in no particular order as
provided by the operating system's underlying directory mechanisms._

## Class: fs.Dirent
<!-- YAML
added: v10.10.0
-->

When [`fs.readdir()`][] or [`fs.readdirSync()`][] is called with the
`withFileTypes` option set to `true`, the resulting array is filled with
A representation of a directory entry, as returned by reading from an [`fs.Dir`][].

Additionally, when [`fs.readdir()`][] or [`fs.readdirSync()`][] is called with
the `withFileTypes` option set to `true`, the resulting array is filled with
`fs.Dirent` objects, rather than strings or `Buffers`.

### dirent.isBlockDevice()
Expand Down Expand Up @@ -2505,6 +2640,46 @@ Returns an integer representing the file descriptor.
For detailed information, see the documentation of the asynchronous version of
this API: [`fs.open()`][].

## fs.opendir(path[, options], callback)
<!-- YAML
added: REPLACEME
-->

* `path` {string|Buffer|URL}
* `options` {Object}
* `encoding` {string|null} **Default:** `'utf8'`
* `callback` {Function}
* `err` {Error}
* `dir` {fs.Dir}

Asynchronously open a directory. See opendir(3).

Creates an [`fs.Dir`][], which contains all further functions for reading from
and cleaning up the directory.

The `encoding` option sets the encoding for the `path` while opening the
directory and subsequent read operations (unless otherwise overriden during
reads from the directory).

## fs.opendirSync(path[, options])
<!-- YAML
added: REPLACEME
-->

* `path` {string|Buffer|URL}
* `options` {Object}
* `encoding` {string|null} **Default:** `'utf8'`
* Returns: {fs.Dir}

Synchronously open a directory. See opendir(3).

Creates an [`fs.Dir`][], which contains all further functions for reading from
and cleaning up the directory.

The `encoding` option sets the encoding for the `path` while opening the
directory and subsequent read operations (unless otherwise overriden during
reads from the directory).

## fs.read(fd, buffer, offset, length, position, callback)
<!-- YAML
added: v0.0.2
Expand Down Expand Up @@ -4643,6 +4818,39 @@ by [Naming Files, Paths, and Namespaces][]. Under NTFS, if the filename contains
a colon, Node.js will open a file system stream, as described by
[this MSDN page][MSDN-Using-Streams].

## fsPromises.opendir(path[, options])
<!-- YAML
added: REPLACEME
-->

* `path` {string|Buffer|URL}
* `options` {Object}
* `encoding` {string|null} **Default:** `'utf8'`
* Returns: {Promise} containing {fs.Dir}

Asynchronously open a directory. See opendir(3).

Creates an [`fs.Dir`][], which contains all further functions for reading from
and cleaning up the directory.

The `encoding` option sets the encoding for the `path` while opening the
directory and subsequent read operations (unless otherwise overriden during
reads from the directory).

Example using async interation:

```js
const fs = require('fs');

async function print(path) {
const dir = await fs.promises.opendir(path);
for await (const dirent of dir) {
console.log(dirent.name);
}
}
print('./').catch(console.error);
```

### fsPromises.readdir(path[, options])
<!-- YAML
added: v10.0.0
Expand Down Expand Up @@ -5252,6 +5460,7 @@ the file contents.
[`UV_THREADPOOL_SIZE`]: cli.html#cli_uv_threadpool_size_size
[`WriteStream`]: #fs_class_fs_writestream
[`event ports`]: https://illumos.org/man/port_create
[`fs.Dir`]: #fs_class_fs_dir
[`fs.Dirent`]: #fs_class_fs_dirent
[`fs.FSWatcher`]: #fs_class_fs_fswatcher
[`fs.Stats`]: #fs_class_fs_stats
Expand All @@ -5268,6 +5477,8 @@ the file contents.
[`fs.mkdir()`]: #fs_fs_mkdir_path_options_callback
[`fs.mkdtemp()`]: #fs_fs_mkdtemp_prefix_options_callback
[`fs.open()`]: #fs_fs_open_path_flags_mode_callback
[`fs.opendir()`]: #fs_fs_opendir_path_options_callback
[`fs.opendirSync()`]: #fs_fs_opendirsync_path_options
[`fs.read()`]: #fs_fs_read_fd_buffer_offset_length_position_callback
[`fs.readFile()`]: #fs_fs_readfile_path_options_callback
[`fs.readFileSync()`]: #fs_fs_readfilesync_path_options
Expand All @@ -5283,6 +5494,7 @@ the file contents.
[`fs.write(fd, string...)`]: #fs_fs_write_fd_string_position_encoding_callback
[`fs.writeFile()`]: #fs_fs_writefile_file_data_options_callback
[`fs.writev()`]: #fs_fs_writev_fd_buffers_position_callback
[`fsPromises.opendir()`]: #fs_fspromises_opendir_path_options
[`inotify(7)`]: http://man7.org/linux/man-pages/man7/inotify.7.html
[`kqueue(2)`]: https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2
[`net.Socket`]: net.html#net_class_net_socket
Expand Down
27 changes: 9 additions & 18 deletions lib/fs.js
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ const {
getDirents,
getOptions,
getValidatedPath,
handleErrorFromBinding,
nullCheck,
preprocessSymlinkDestination,
Stats,
Expand All @@ -79,6 +80,11 @@ const {
validateRmdirOptions,
warnOnNonPortableTemplate
} = require('internal/fs/utils');
const {
Dir,
opendir,
opendirSync
} = require('internal/fs/dir');
const {
CHAR_FORWARD_SLASH,
CHAR_BACKWARD_SLASH,
Expand Down Expand Up @@ -122,23 +128,6 @@ function showTruncateDeprecation() {
}
}

function handleErrorFromBinding(ctx) {
if (ctx.errno !== undefined) { // libuv error numbers
const err = uvException(ctx);
// eslint-disable-next-line no-restricted-syntax
Error.captureStackTrace(err, handleErrorFromBinding);
throw err;
}
if (ctx.error !== undefined) { // Errors created in C++ land.
// TODO(joyeecheung): currently, ctx.error are encoding errors
// usually caused by memory problems. We need to figure out proper error
// code(s) for this.
// eslint-disable-next-line no-restricted-syntax
Error.captureStackTrace(ctx.error, handleErrorFromBinding);
throw ctx.error;
}
}

function maybeCallback(cb) {
if (typeof cb === 'function')
return cb;
Expand Down Expand Up @@ -1834,7 +1823,6 @@ function createWriteStream(path, options) {
return new WriteStream(path, options);
}


module.exports = fs = {
appendFile,
appendFileSync,
Expand Down Expand Up @@ -1880,6 +1868,8 @@ module.exports = fs = {
mkdtempSync,
open,
openSync,
opendir,
opendirSync,
readdir,
readdirSync,
read,
Expand Down Expand Up @@ -1913,6 +1903,7 @@ module.exports = fs = {
writeSync,
writev,
writevSync,
Dir,
Dirent,
Stats,

Expand Down
1 change: 1 addition & 0 deletions lib/internal/errors.js
Original file line number Diff line number Diff line change
Expand Up @@ -756,6 +756,7 @@ E('ERR_CRYPTO_SCRYPT_NOT_SUPPORTED', 'Scrypt algorithm not supported', Error);
E('ERR_CRYPTO_SIGN_KEY_REQUIRED', 'No key provided to sign', Error);
E('ERR_CRYPTO_TIMING_SAFE_EQUAL_LENGTH',
'Input buffers must have the same byte length', RangeError);
E('ERR_DIR_CLOSED', 'Directory handle was closed', Error);
E('ERR_DNS_SET_SERVERS_FAILED', 'c-ares failed to set servers: "%s" [%s]',
Error);
E('ERR_DOMAIN_CALLBACK_NOT_AVAILABLE',
Expand Down
Loading

0 comments on commit 91561d0

Please sign in to comment.