[RFC] Provide a file system API


Thomas, Ramesh
 

A file system is necessary to store data reliably and to be accessed in
a consistent and deterministic manner. Example applications are data
logging, media files and file based databases which would need to be
stored for later processing off-line or to be sent over radio in bursts
instead of per sample captured.

Related requirement:
https://jira.zephyrproject.org/browse/ZEP-228

***Proposal***
The Zephyr File System API will follow interfaces similar to stdio and
POSIX so it is familiar to users. However, it does not claim to be
compliant to POSIX or any other specification.

One of the goals is to keep clean separation of the file API layer and
the disk interface. Following diagram illustrates the layer separation:

+--------------------------+
| Zephyr File System API |
+--------------------------+
| Glue layer for FS module |
+--------------------------+
| File System module |
+--------------------------+
| Disk I/O interface |
+--------------------------+
| Storage media interface |
+--------------------------+

The File system module would be an implementation that provides clean
separation between the API layer and disk I/O. A glue layer will
abstract its own interface to that of the Zephyr FS API. The disk i/o
interface will use Zephyr's flash HAL.

***The File System API descriptions follow***
Gerrit link:
https://gerrit.zephyrproject.org/r/#/c/3402/9/include/fs.h

/**
* @brief File open
*
* Opens an existing file or create a new one and associates
* a stream with it.
*
* @param zfp Pointer to file object
* @param file_name The name of file to open
* @param mode Flags indicating open mode
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_open(ZFILE *zfp, const char *file_name, const char *mode);

/**
* @brief File close
*
* Flushes the associated stream and closes
* the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_close(ZFILE *zfp);

/**
* @brief File unlink
*
* Deletes the specified file or directory
*
* @param path Path to the file or directory to delete
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_unlink(const char *path);

/**
* @brief File read
*
* Reads items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be read
*
* @return Number of bytes read. On success, it will be equal to number of
* items requested to be read. Returns less than number of bytes
* requested if there are not enough bytes available in file. Will return
* -ERRNO code on error.
*/
ssize_t fs_read(ZFILE *zfp, void *ptr, size_t size);

/**
* @brief File write
*
* Writes items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be write
*
* @return Number of bytes written. On success, it will be equal to
number of
* bytes requested to be written. Will return -ERRNO code on error.
*/
ssize_t fs_write(ZFILE *zfp, void *ptr, size_t size);

/**
* @brief File seek
*
* Moves the file position to a new location in the file. The offset is
added
* to file position based on the 'from' parameter.
*
* @param zfp Pointer to the file object
* @param offset Relative location to move the file pointer to
* @param from Relative location from where offset is to be calculated.
* SEEK_SET = from beginning of file
* SEE_CUR = from current position,
* SEEK_END = from end of file.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_seek(ZFILE *zfp, long offset, int from);


/**
* @brief Get current file position.
*
* Retrieves the current position in the file.
*
* @param zfp Pointer to the file object
*
* @retval position Current position in file
* Current revision does not validate the file object.
*/
long fs_tell(ZFILE *zfp);

/**
* @brief Get last error
*
* Retrieves the last error that occurred for the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 if no error had occurred
* @retval -ERRNO errno code if error had occurred.
* Current revision does not validate the file object.
*/
int fs_error(ZFILE *zfp);

/**
* @brief Tests if EOF indicator is set for the file.
*
* Retrieves the EOF status of the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 if not EOF
* @retval Non zero if EOF
* Current revision does not validate the file object.
*/
int fs_eof(ZFILE *zfp);

/**
* @brief Get file size
*
* Retrieves the size of the file.
*
* @param zfp Pointer to the file object
*
* @retval size of the file in bytes
* Current revision does not validate the file object.
*/
long fs_size(ZFILE *zfp);

/**
* @brief File system init
*
* Application calls this to initialize the FS interface. This should
* be the first function the application should call if it will be
* using the file system.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_init(void);

/**
* @brief Creates and formats a volume
*
* Formats the storage media with underlying file system. This will
* typically be used by a provisioning tool during build or flashing
* of OS image. The file system configurations will be taken from
* Kconfig settings.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_format(void);

/**
* @brief Mounts a volume
*
* Mounts an existing file system. The volume configurations
* are taken from Kconfig settings.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_mount(void);

/**
* @brief Directory create
*
* Creates a new directory using specified path.
*
* @param path Path to the directory to create
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_mkdir(const char *path);

/**
* @brief Directory open
*
* Opens an existing directory specified by the path.
*
* @param zdp Pointer to the directory object
* @param path Path to the directory to open
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_opendir(ZDIR *zdp, const char *path);

/**
* @brief Directory read entry
*
* Reads directory entries of a open directory
*
* @param zdp Pointer to the directory object
* @param entry Pointer to dir_entry structure to read the entry into
* struct dir_entry will store following:
* attrib : Whether it is a file or directory
* name : name of file or directory
* size : size of file
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_readdir(ZDIR *zdp, struct dir_entry *entry);

/**
* @brief Directory close
*
* Closes an open directory.
*
* @param zdp Pointer to the directory object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_closedir(ZDIR *zdp);


Mitsis, Peter <Peter.Mitsis@...>
 

See [Peter]

-----Original Message-----
From: Ramesh Thomas [mailto:ramesh.thomas(a)intel.com]
Sent: July-26-16 9:37 PM
To: devel(a)lists.zephyrproject.org
Subject: [devel] [RFC] Provide a file system API

A file system is necessary to store data reliably and to be accessed in a
consistent and deterministic manner. Example applications are data
logging, media files and file based databases which would need to be
stored for later processing off-line or to be sent over radio in bursts
instead of per sample captured.

Related requirement:
https://jira.zephyrproject.org/browse/ZEP-228

***Proposal***
The Zephyr File System API will follow interfaces similar to stdio and
POSIX so it is familiar to users. However, it does not claim to be
compliant to POSIX or any other specification.

One of the goals is to keep clean separation of the file API layer and the
disk interface. Following diagram illustrates the layer separation:

+--------------------------+
| Zephyr File System API |
+--------------------------+
| Glue layer for FS module |
+--------------------------+
| File System module |
+--------------------------+
| Disk I/O interface |
+--------------------------+
| Storage media interface |
+--------------------------+

The File system module would be an implementation that provides clean
separation between the API layer and disk I/O. A glue layer will abstract
its own interface to that of the Zephyr FS API. The disk i/o interface
will use Zephyr's flash HAL.

***The File System API descriptions follow*** Gerrit link:
https://gerrit.zephyrproject.org/r/#/c/3402/9/include/fs.h

/**
* @brief File open
*
* Opens an existing file or create a new one and associates
* a stream with it.
*
* @param zfp Pointer to file object
* @param file_name The name of file to open
* @param mode Flags indicating open mode
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_open(ZFILE *zfp, const char *file_name, const char *mode);

/**
* @brief File close
*
* Flushes the associated stream and closes
* the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_close(ZFILE *zfp);

/**
* @brief File unlink
*
* Deletes the specified file or directory
*
* @param path Path to the file or directory to delete
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_unlink(const char *path);

/**
* @brief File read
*
* Reads items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be read
*
* @return Number of bytes read. On success, it will be equal to number of
* items requested to be read. Returns less than number of bytes
* requested if there are not enough bytes available in file. Will return
* -ERRNO code on error.
*/
ssize_t fs_read(ZFILE *zfp, void *ptr, size_t size);

/**
* @brief File write
*
* Writes items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be write
*
* @return Number of bytes written. On success, it will be equal to number
of
* bytes requested to be written. Will return -ERRNO code on error.
*/
ssize_t fs_write(ZFILE *zfp, void *ptr, size_t size);

/**
* @brief File seek
*
* Moves the file position to a new location in the file. The offset is
added
* to file position based on the 'from' parameter.
*
* @param zfp Pointer to the file object
* @param offset Relative location to move the file pointer to
* @param from Relative location from where offset is to be calculated.
* SEEK_SET = from beginning of file
* SEE_CUR = from current position,
* SEEK_END = from end of file.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_seek(ZFILE *zfp, long offset, int from);


/**
* @brief Get current file position.
*
* Retrieves the current position in the file.
*
* @param zfp Pointer to the file object
*
* @retval position Current position in file
* Current revision does not validate the file object.
*/
long fs_tell(ZFILE *zfp);

/**
* @brief Get last error
*
* Retrieves the last error that occurred for the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 if no error had occurred
* @retval -ERRNO errno code if error had occurred.
* Current revision does not validate the file object.
*/
int fs_error(ZFILE *zfp);

/**
* @brief Tests if EOF indicator is set for the file.
*
* Retrieves the EOF status of the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 if not EOF
* @retval Non zero if EOF
* Current revision does not validate the file object.
*/
int fs_eof(ZFILE *zfp);

/**
* @brief Get file size
*
* Retrieves the size of the file.
*
* @param zfp Pointer to the file object
*
* @retval size of the file in bytes
* Current revision does not validate the file object.
*/
long fs_size(ZFILE *zfp);

/**
* @brief File system init
*
* Application calls this to initialize the FS interface. This should
* be the first function the application should call if it will be
* using the file system.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_init(void);

/**
* @brief Creates and formats a volume
*
* Formats the storage media with underlying file system. This will
* typically be used by a provisioning tool during build or flashing
* of OS image. The file system configurations will be taken from
* Kconfig settings.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_format(void);

/**
* @brief Mounts a volume
*
* Mounts an existing file system. The volume configurations
* are taken from Kconfig settings.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_mount(void);

/**
* @brief Directory create
*
* Creates a new directory using specified path.
*
* @param path Path to the directory to create
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_mkdir(const char *path);

/**
* @brief Directory open
*
* Opens an existing directory specified by the path.
*
* @param zdp Pointer to the directory object
* @param path Path to the directory to open
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_opendir(ZDIR *zdp, const char *path);

/**
* @brief Directory read entry
*
* Reads directory entries of a open directory
*
* @param zdp Pointer to the directory object
* @param entry Pointer to dir_entry structure to read the entry into
* struct dir_entry will store following:
* attrib : Whether it is a file or directory
* name : name of file or directory
* size : size of file
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_readdir(ZDIR *zdp, struct dir_entry *entry);
[Peter] - Should we be using a re-entrant version? That is something with a signature more like readdir_r()? (My apologies if this has already been discussed.)


/**
* @brief Directory close
*
* Closes an open directory.
*
* @param zdp Pointer to the directory object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_closedir(ZDIR *zdp);
[Peter]
I see the following functionality as missing. Is this intentional?
1. Some method to truncate/grow/shrink a file.
2. Some method to query the FS for stats such number of unused blocks


Andy Ross
 

Some random kibitzing. I'm new, so apologies if some of this has been
discussed before:

int fs_open(ZFILE *zfp, const char *file_name, const char *mode);
Why a stdio-style mode string that has to be parsed and not a
fcntl-style flag word which seems like a better fit for the OS? And
to be serious: do we even care about mode? It seems like runtime
checking that you aren't writing to a read-only descriptor is maybe
senseless complexity. And what's the use case for the OS checking
file permissions on Zephyr?

int fs_error(ZFILE *zfp);
What's the use case for an ferror() equivalent? This is a hack in the
C library to work around edge cases of errno not being convenient to
check across multiple I/O calls when the underlying thing might be
something other than a disk. Can't you just arrange things here such
that all functions return a synchronous error code (a quick scan says
you already have) and skip this?

int fs_eof(ZFILE *zfp);
Likewise this doesn't seem useful in an API where everything is known
to be a disk file. Just check fs_tell(p)>=fs_size(p). Though I guess
if that's the implementation internally then it's sane enough just to
leave this as a convenience function.

long fs_size(ZFILE *zfp);
Should return size_t (or ssize_t if an error is possible, I guess).

int fs_init(void);
Do we really need this? There are other initialization hooks you can
use like mount and open. Why must there be a global one?

int fs_format(void);
That doesn't look useful as-is. Even FAT32 has a bunch of options you
are going to want to set in a real-world application (filesystem
label, cluster size, etc...). Realistically maybe this wants to be
provided by a lower-level filesystem driver and not an abstraction
layer? (Which is the way unix does it -- mkfs sits on top of the
block device and not stdio). Or at least pass a string or something
for configuration data that the filesystem can interpret.

int fs_mount(void);
Similarly this wants more context if it's going to be more than a toy.
Right now it's just a synonym for fs_init(), right? There's no
practical difference detectable to a user between the states, they
just need to blindly call two functions with no arguments if they want
"filesystem". And in the future they'll always need to do so,
blindly, because there's nothing else the API gives them. Real apps
are going to want to do things like detect a card insertion and
remount, have more than one filesystem, etc... And this won't evolve
in that direction without an API change.

If you're going to have this, at least pass some configuration
(e.g. three strings for device/directory/options, etc...) to make it
futureproof.

int fs_readdir(ZDIR *zdp, struct dir_entry *entry);
Where's the definition for stuct dir_entry?

Andy


Thomas, Ramesh
 

On 07/27/2016 10:03 AM, Mitsis, Peter wrote:
See [Peter]

-----Original Message-----
From: Ramesh Thomas [mailto:ramesh.thomas(a)intel.com]
Sent: July-26-16 9:37 PM
To: devel(a)lists.zephyrproject.org
Subject: [devel] [RFC] Provide a file system API

A file system is necessary to store data reliably and to be accessed in a
consistent and deterministic manner. Example applications are data
logging, media files and file based databases which would need to be
stored for later processing off-line or to be sent over radio in bursts
instead of per sample captured.

Related requirement:
https://jira.zephyrproject.org/browse/ZEP-228

***Proposal***
The Zephyr File System API will follow interfaces similar to stdio and
POSIX so it is familiar to users. However, it does not claim to be
compliant to POSIX or any other specification.

One of the goals is to keep clean separation of the file API layer and the
disk interface. Following diagram illustrates the layer separation:

+--------------------------+
| Zephyr File System API |
+--------------------------+
| Glue layer for FS module |
+--------------------------+
| File System module |
+--------------------------+
| Disk I/O interface |
+--------------------------+
| Storage media interface |
+--------------------------+

The File system module would be an implementation that provides clean
separation between the API layer and disk I/O. A glue layer will abstract
its own interface to that of the Zephyr FS API. The disk i/o interface
will use Zephyr's flash HAL.

***The File System API descriptions follow*** Gerrit link:
https://gerrit.zephyrproject.org/r/#/c/3402/9/include/fs.h

/**
* @brief File open
*
* Opens an existing file or create a new one and associates
* a stream with it.
*
* @param zfp Pointer to file object
* @param file_name The name of file to open
* @param mode Flags indicating open mode
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_open(ZFILE *zfp, const char *file_name, const char *mode);

/**
* @brief File close
*
* Flushes the associated stream and closes
* the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_close(ZFILE *zfp);

/**
* @brief File unlink
*
* Deletes the specified file or directory
*
* @param path Path to the file or directory to delete
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_unlink(const char *path);

/**
* @brief File read
*
* Reads items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be read
*
* @return Number of bytes read. On success, it will be equal to number of
* items requested to be read. Returns less than number of bytes
* requested if there are not enough bytes available in file. Will return
* -ERRNO code on error.
*/
ssize_t fs_read(ZFILE *zfp, void *ptr, size_t size);

/**
* @brief File write
*
* Writes items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be write
*
* @return Number of bytes written. On success, it will be equal to number
of
* bytes requested to be written. Will return -ERRNO code on error.
*/
ssize_t fs_write(ZFILE *zfp, void *ptr, size_t size);

/**
* @brief File seek
*
* Moves the file position to a new location in the file. The offset is
added
* to file position based on the 'from' parameter.
*
* @param zfp Pointer to the file object
* @param offset Relative location to move the file pointer to
* @param from Relative location from where offset is to be calculated.
* SEEK_SET = from beginning of file
* SEE_CUR = from current position,
* SEEK_END = from end of file.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_seek(ZFILE *zfp, long offset, int from);


/**
* @brief Get current file position.
*
* Retrieves the current position in the file.
*
* @param zfp Pointer to the file object
*
* @retval position Current position in file
* Current revision does not validate the file object.
*/
long fs_tell(ZFILE *zfp);

/**
* @brief Get last error
*
* Retrieves the last error that occurred for the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 if no error had occurred
* @retval -ERRNO errno code if error had occurred.
* Current revision does not validate the file object.
*/
int fs_error(ZFILE *zfp);

/**
* @brief Tests if EOF indicator is set for the file.
*
* Retrieves the EOF status of the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 if not EOF
* @retval Non zero if EOF
* Current revision does not validate the file object.
*/
int fs_eof(ZFILE *zfp);

/**
* @brief Get file size
*
* Retrieves the size of the file.
*
* @param zfp Pointer to the file object
*
* @retval size of the file in bytes
* Current revision does not validate the file object.
*/
long fs_size(ZFILE *zfp);

/**
* @brief File system init
*
* Application calls this to initialize the FS interface. This should
* be the first function the application should call if it will be
* using the file system.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_init(void);

/**
* @brief Creates and formats a volume
*
* Formats the storage media with underlying file system. This will
* typically be used by a provisioning tool during build or flashing
* of OS image. The file system configurations will be taken from
* Kconfig settings.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_format(void);

/**
* @brief Mounts a volume
*
* Mounts an existing file system. The volume configurations
* are taken from Kconfig settings.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_mount(void);

/**
* @brief Directory create
*
* Creates a new directory using specified path.
*
* @param path Path to the directory to create
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_mkdir(const char *path);

/**
* @brief Directory open
*
* Opens an existing directory specified by the path.
*
* @param zdp Pointer to the directory object
* @param path Path to the directory to open
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_opendir(ZDIR *zdp, const char *path);

/**
* @brief Directory read entry
*
* Reads directory entries of a open directory
*
* @param zdp Pointer to the directory object
* @param entry Pointer to dir_entry structure to read the entry into
* struct dir_entry will store following:
* attrib : Whether it is a file or directory
* name : name of file or directory
* size : size of file
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_readdir(ZDIR *zdp, struct dir_entry *entry);
[Peter] - Should we be using a re-entrant version? That is something with a signature more like readdir_r()? (My apologies if this has already been discussed.)
We don't have a non-re-entrant version so we don't need to differentiate
the way stdio needs to. Also we are not strictly following stdio signature.



/**
* @brief Directory close
*
* Closes an open directory.
*
* @param zdp Pointer to the directory object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_closedir(ZDIR *zdp);
[Peter]
I see the following functionality as missing. Is this intentional?
1. Some method to truncate/grow/shrink a file.
2. Some method to query the FS for stats such number of unused blocks
These functions can be useful and we can add them.


Thomas, Ramesh
 

On 07/27/2016 11:10 AM, Andy Ross wrote:
Some random kibitzing. I'm new, so apologies if some of this has been
discussed before:
No, they have not been discussed. Thanks for your feedbacks.


int fs_open(ZFILE *zfp, const char *file_name, const char *mode);
Why a stdio-style mode string that has to be parsed and not a
fcntl-style flag word which seems like a better fit for the OS? And
to be serious: do we even care about mode? It seems like runtime
checking that you aren't writing to a read-only descriptor is maybe
senseless complexity. And what's the use case for the OS checking
file permissions on Zephyr?
I totally agree with you. Following are couple of options.

1. int fs_open(ZFILE *zfp, const char *file_name);
- Opens file if exists
- Creates a new one if file does not exist

2. int fs_open(ZFILE *zfp, const char *file_name);
- Opens file if exists
- Returns error if file does not exist
int fs_create(ZFILE *zfp, const char *file_name);
- Create file always.
- Will truncate file if one with same name exists.

First one seems simpler to me.


int fs_error(ZFILE *zfp);
What's the use case for an ferror() equivalent? This is a hack in the
C library to work around edge cases of errno not being convenient to
check across multiple I/O calls when the underlying thing might be
something other than a disk. Can't you just arrange things here such
that all functions return a synchronous error code (a quick scan says
you already have) and skip this?
Agree, there is no use case. Will remove.


int fs_eof(ZFILE *zfp);
Likewise this doesn't seem useful in an API where everything is known
to be a disk file. Just check fs_tell(p)>=fs_size(p). Though I guess
if that's the implementation internally then it's sane enough just to
leave this as a convenience function.
I don't see any use case for this. Both fs_read and fs_write return
enough information to identify eof condition (vs error). Will remove.


long fs_size(ZFILE *zfp);
Should return size_t (or ssize_t if an error is possible, I guess).
Will change to size_t.


int fs_init(void);
Do we really need this? There are other initialization hooks you can
use like mount and open. Why must there be a global one?

int fs_format(void);
That doesn't look useful as-is. Even FAT32 has a bunch of options you
are going to want to set in a real-world application (filesystem
label, cluster size, etc...). Realistically maybe this wants to be
provided by a lower-level filesystem driver and not an abstraction
layer? (Which is the way unix does it -- mkfs sits on top of the
block device and not stdio). Or at least pass a string or something
for configuration data that the filesystem can interpret.

int fs_mount(void);
Similarly this wants more context if it's going to be more than a toy.
Right now it's just a synonym for fs_init(), right? There's no
practical difference detectable to a user between the states, they
just need to blindly call two functions with no arguments if they want
"filesystem". And in the future they'll always need to do so,
blindly, because there's nothing else the API gives them. Real apps
are going to want to do things like detect a card insertion and
remount, have more than one filesystem, etc... And this won't evolve
in that direction without an API change.

If you're going to have this, at least pass some configuration
(e.g. three strings for device/directory/options, etc...) to make it
futureproof.
I can see your point here that these functions do not meet the
requirement of an API. These need not be exported and should be done
internally.

I would need one function which the user would need to call where all
these can be done. That can be done inside fs_init().


int fs_readdir(ZDIR *zdp, struct dir_entry *entry);
Where's the definition for stuct dir_entry?
enum dir_entry_type {
dir_entry_file,
dir_entry_dir
};

struct dir_entry {
enum dir_entry_type type; /* Whether file or directory */
char *name; /* Name of directory or file */
size_t size; /* Size of file. 0 if directory */
};


Andy




Thomas, Ramesh
 

On 07/26/2016 06:36 PM, Ramesh Thomas wrote:


/**
* @brief File seek
*
* Moves the file position to a new location in the file. The offset is
added
* to file position based on the 'from' parameter.
*
* @param zfp Pointer to the file object
* @param offset Relative location to move the file pointer to
* @param from Relative location from where offset is to be calculated.
* SEEK_SET = from beginning of file
* SEE_CUR = from current position,
* SEEK_END = from end of file.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_seek(ZFILE *zfp, long offset, int from);
I would like to simplify this and always move the pointer relative to
beginning of file. The other options can be easily accomplished by
ftell() + offset and fsize() + offset, if necessary.

/**
* @brief File seek
*
* Moves the file position to a new location in the file.
*
* @param zfp Pointer to the file object
* @param offset New location to move the file pointer to
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_seek(ZFILE *zfp, long offset);

-Ramesh


Thomas, Ramesh
 

Thanks for all the feedbacks. The APIs, after updating with feedbacks
from the mailing list and gerrit, are listed below. Will track and
submit updates if any comments were not addressed correctly.

Currently there is one Jira item created to address Peter's comments
https://jira.zephyrproject.org/browse/ZEP-622

Gerrit link:
https://gerrit.zephyrproject.org/r/#/c/3402/

***API descriptions follow***

/**
* @brief File open
*
* Opens an existing file or create a new one and associates
* a stream with it.
*
* @param zfp Pointer to file object
* @param file_name The name of file to open
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_open(ZFILE *zfp, const char *file_name);

/**
* @brief File close
*
* Flushes the associated stream and closes
* the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_close(ZFILE *zfp);

/**
* @brief File unlink
*
* Deletes the specified file or directory
*
* @param path Path to the file or directory to delete
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_unlink(const char *path);

/**
* @brief File read
*
* Reads items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be read
*
* @return Number of bytes read. On success, it will be equal to number of
* items requested to be read. Returns less than number of bytes
* requested if there are not enough bytes available in file. Will return
* -ERRNO code on error.
*/
ssize_t fs_read(ZFILE *zfp, void *ptr, size_t size);

/**
* @brief File write
*
* Writes items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be write
*
* @return Number of bytes written. On success, it will be equal to the
number
* of bytes requested to be written. Any other value, indicates an
error. Will
* return -ERRNO code on error.
* In the case where -ERRNO is returned, the file pointer will not be
* advanced because it couldn't start the operation.
* In the case where it is able to write, but is not able to complete
writing
* all of the requested number of bytes, then it is because the disk got
full.
* In that case, it returns less number of bytes written than requested, but
* not a negative -ERRNO value as in regular error case.
*/
ssize_t fs_write(ZFILE *zfp, const void *ptr, size_t size);

/**
* @brief File seek
*
* Moves the file position to a new location in the file. The offset is
added
* to file position based on the 'whence' parameter.
*
* @param zfp Pointer to the file object
* @param offset Relative location to move the file pointer to
* @param whence Relative location from where offset is to be calculated.
* SEEK_SET = from beginning of file
* SEE_CUR = from current position,
* SEEK_END = from end of file.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_seek(ZFILE *zfp, off_t offset, int whence);

/**
* @brief Get current file position.
*
* Retrieves the current position in the file.
*
* @param zfp Pointer to the file object
*
* @retval position Current position in file
* Current revision does not validate the file object.
*/
off_t fs_tell(ZFILE *zfp);

/**
* @brief Directory create
*
* Creates a new directory using specified path.
*
* @param path Path to the directory to create
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_mkdir(const char *path);

/**
* @brief Directory open
*
* Opens an existing directory specified by the path.
*
* @param zdp Pointer to the directory object
* @param path Path to the directory to open
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_opendir(ZDIR *zdp, const char *path);

enum dir_entry_type {
DIR_ENTRY_FILE,
DIR_ENTRY_DIR
};

struct zfs_dirent {
enum dir_entry_type type; /* Whether file or directory */
char name[MAX_FILE_NAME + 1]; /* Name of directory or file */
size_t size; /* Size of file. 0 if directory */
};

/**
* @brief Directory read entry
*
* Reads directory entries of a open directory
*
* @param zdp Pointer to the directory object
* @param entry Pointer to zfs_dirent structure to read the entry into
*
* @retval 0 Success
* @retval -ERRNO errno code if error
* @return In end-of-dir condition, this will return 0 and set
* entry->name[0] = 0
*/
int fs_readdir(ZDIR *zdp, struct zfs_dirent *entry);

/**
* @brief Directory close
*
* Closes an open directory.
*
* @param zdp Pointer to the directory object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_closedir(ZDIR *zdp);

/**
* @brief File or directory status
*
* Checks the status of a file or directory specified by the path
*
* @param path Path to the file or directory
* @param entry Pointer to zfs_dirent structure to fill if file or directory
* exists.
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_stat(const char *path, struct zfs_dirent *entry);


Kumar Gala
 

Is there a reason to not just use the stdlib FILE interfaces?

I’m not sure I understand the reason to not just use FILE instead of ZFILE.

- k

On Aug 1, 2016, at 11:07 PM, Ramesh Thomas <ramesh.thomas(a)intel.com> wrote:


Thanks for all the feedbacks. The APIs, after updating with feedbacks
from the mailing list and gerrit, are listed below. Will track and
submit updates if any comments were not addressed correctly.

Currently there is one Jira item created to address Peter's comments
https://jira.zephyrproject.org/browse/ZEP-622

Gerrit link:
https://gerrit.zephyrproject.org/r/#/c/3402/

***API descriptions follow***

/**
* @brief File open
*
* Opens an existing file or create a new one and associates
* a stream with it.
*
* @param zfp Pointer to file object
* @param file_name The name of file to open
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_open(ZFILE *zfp, const char *file_name);

/**
* @brief File close
*
* Flushes the associated stream and closes
* the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_close(ZFILE *zfp);

/**
* @brief File unlink
*
* Deletes the specified file or directory
*
* @param path Path to the file or directory to delete
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_unlink(const char *path);

/**
* @brief File read
*
* Reads items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be read
*
* @return Number of bytes read. On success, it will be equal to number of
* items requested to be read. Returns less than number of bytes
* requested if there are not enough bytes available in file. Will return
* -ERRNO code on error.
*/
ssize_t fs_read(ZFILE *zfp, void *ptr, size_t size);

/**
* @brief File write
*
* Writes items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be write
*
* @return Number of bytes written. On success, it will be equal to the
number
* of bytes requested to be written. Any other value, indicates an
error. Will
* return -ERRNO code on error.
* In the case where -ERRNO is returned, the file pointer will not be
* advanced because it couldn't start the operation.
* In the case where it is able to write, but is not able to complete
writing
* all of the requested number of bytes, then it is because the disk got
full.
* In that case, it returns less number of bytes written than requested, but
* not a negative -ERRNO value as in regular error case.
*/
ssize_t fs_write(ZFILE *zfp, const void *ptr, size_t size);

/**
* @brief File seek
*
* Moves the file position to a new location in the file. The offset is
added
* to file position based on the 'whence' parameter.
*
* @param zfp Pointer to the file object
* @param offset Relative location to move the file pointer to
* @param whence Relative location from where offset is to be calculated.
* SEEK_SET = from beginning of file
* SEE_CUR = from current position,
* SEEK_END = from end of file.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_seek(ZFILE *zfp, off_t offset, int whence);

/**
* @brief Get current file position.
*
* Retrieves the current position in the file.
*
* @param zfp Pointer to the file object
*
* @retval position Current position in file
* Current revision does not validate the file object.
*/
off_t fs_tell(ZFILE *zfp);

/**
* @brief Directory create
*
* Creates a new directory using specified path.
*
* @param path Path to the directory to create
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_mkdir(const char *path);

/**
* @brief Directory open
*
* Opens an existing directory specified by the path.
*
* @param zdp Pointer to the directory object
* @param path Path to the directory to open
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_opendir(ZDIR *zdp, const char *path);

enum dir_entry_type {
DIR_ENTRY_FILE,
DIR_ENTRY_DIR
};

struct zfs_dirent {
enum dir_entry_type type; /* Whether file or directory */
char name[MAX_FILE_NAME + 1]; /* Name of directory or file */
size_t size; /* Size of file. 0 if directory */
};

/**
* @brief Directory read entry
*
* Reads directory entries of a open directory
*
* @param zdp Pointer to the directory object
* @param entry Pointer to zfs_dirent structure to read the entry into
*
* @retval 0 Success
* @retval -ERRNO errno code if error
* @return In end-of-dir condition, this will return 0 and set
* entry->name[0] = 0
*/
int fs_readdir(ZDIR *zdp, struct zfs_dirent *entry);

/**
* @brief Directory close
*
* Closes an open directory.
*
* @param zdp Pointer to the directory object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_closedir(ZDIR *zdp);

/**
* @brief File or directory status
*
* Checks the status of a file or directory specified by the path
*
* @param path Path to the file or directory
* @param entry Pointer to zfs_dirent structure to fill if file or directory
* exists.
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_stat(const char *path, struct zfs_dirent *entry);


Thomas, Ramesh
 

On 8/2/2016 6:58 AM, Kumar Gala wrote:
Is there a reason to not just use the stdlib FILE interfaces?
The primary goal is to provide a file system without impacting the small
footprint requirements of the OS. Using some existing interface would
force the design to be fully compliant with that interface, which will
cause unnecessary overheads.


I’m not sure I understand the reason to not just use FILE instead of ZFILE.
It would conflict with the definition of FILE in stdio. The file system
does not replace the implementation of FILE type defined in stdio.


- k

On Aug 1, 2016, at 11:07 PM, Ramesh Thomas <ramesh.thomas(a)intel.com> wrote:


Thanks for all the feedbacks. The APIs, after updating with feedbacks
from the mailing list and gerrit, are listed below. Will track and
submit updates if any comments were not addressed correctly.

Currently there is one Jira item created to address Peter's comments
https://jira.zephyrproject.org/browse/ZEP-622

Gerrit link:
https://gerrit.zephyrproject.org/r/#/c/3402/

***API descriptions follow***

/**
* @brief File open
*
* Opens an existing file or create a new one and associates
* a stream with it.
*
* @param zfp Pointer to file object
* @param file_name The name of file to open
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_open(ZFILE *zfp, const char *file_name);

/**
* @brief File close
*
* Flushes the associated stream and closes
* the file.
*
* @param zfp Pointer to the file object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_close(ZFILE *zfp);

/**
* @brief File unlink
*
* Deletes the specified file or directory
*
* @param path Path to the file or directory to delete
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_unlink(const char *path);

/**
* @brief File read
*
* Reads items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be read
*
* @return Number of bytes read. On success, it will be equal to number of
* items requested to be read. Returns less than number of bytes
* requested if there are not enough bytes available in file. Will return
* -ERRNO code on error.
*/
ssize_t fs_read(ZFILE *zfp, void *ptr, size_t size);

/**
* @brief File write
*
* Writes items of data of size bytes long.
*
* @param zfp Pointer to the file object
* @param ptr Pointer to the data buffer
* @param size Number of bytes to be write
*
* @return Number of bytes written. On success, it will be equal to the
number
* of bytes requested to be written. Any other value, indicates an
error. Will
* return -ERRNO code on error.
* In the case where -ERRNO is returned, the file pointer will not be
* advanced because it couldn't start the operation.
* In the case where it is able to write, but is not able to complete
writing
* all of the requested number of bytes, then it is because the disk got
full.
* In that case, it returns less number of bytes written than requested, but
* not a negative -ERRNO value as in regular error case.
*/
ssize_t fs_write(ZFILE *zfp, const void *ptr, size_t size);

/**
* @brief File seek
*
* Moves the file position to a new location in the file. The offset is
added
* to file position based on the 'whence' parameter.
*
* @param zfp Pointer to the file object
* @param offset Relative location to move the file pointer to
* @param whence Relative location from where offset is to be calculated.
* SEEK_SET = from beginning of file
* SEE_CUR = from current position,
* SEEK_END = from end of file.
*
* @retval 0 Success
* @retval -ERRNO errno code if error.
*/
int fs_seek(ZFILE *zfp, off_t offset, int whence);

/**
* @brief Get current file position.
*
* Retrieves the current position in the file.
*
* @param zfp Pointer to the file object
*
* @retval position Current position in file
* Current revision does not validate the file object.
*/
off_t fs_tell(ZFILE *zfp);

/**
* @brief Directory create
*
* Creates a new directory using specified path.
*
* @param path Path to the directory to create
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_mkdir(const char *path);

/**
* @brief Directory open
*
* Opens an existing directory specified by the path.
*
* @param zdp Pointer to the directory object
* @param path Path to the directory to open
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_opendir(ZDIR *zdp, const char *path);

enum dir_entry_type {
DIR_ENTRY_FILE,
DIR_ENTRY_DIR
};

struct zfs_dirent {
enum dir_entry_type type; /* Whether file or directory */
char name[MAX_FILE_NAME + 1]; /* Name of directory or file */
size_t size; /* Size of file. 0 if directory */
};

/**
* @brief Directory read entry
*
* Reads directory entries of a open directory
*
* @param zdp Pointer to the directory object
* @param entry Pointer to zfs_dirent structure to read the entry into
*
* @retval 0 Success
* @retval -ERRNO errno code if error
* @return In end-of-dir condition, this will return 0 and set
* entry->name[0] = 0
*/
int fs_readdir(ZDIR *zdp, struct zfs_dirent *entry);

/**
* @brief Directory close
*
* Closes an open directory.
*
* @param zdp Pointer to the directory object
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_closedir(ZDIR *zdp);

/**
* @brief File or directory status
*
* Checks the status of a file or directory specified by the path
*
* @param path Path to the file or directory
* @param entry Pointer to zfs_dirent structure to fill if file or directory
* exists.
*
* @retval 0 Success
* @retval -ERRNO errno code if error
*/
int fs_stat(const char *path, struct zfs_dirent *entry);


Benjamin Walsh <benjamin.walsh@...>
 

On Tue, Jul 26, 2016 at 06:36:45PM -0700, Ramesh Thomas wrote:
A file system is necessary to store data reliably and to be accessed in
a consistent and deterministic manner. Example applications are data
logging, media files and file based databases which would need to be
stored for later processing off-line or to be sent over radio in bursts
instead of per sample captured.

Related requirement:
https://jira.zephyrproject.org/browse/ZEP-228

***Proposal***
The Zephyr File System API will follow interfaces similar to stdio and
POSIX so it is familiar to users. However, it does not claim to be
compliant to POSIX or any other specification.

One of the goals is to keep clean separation of the file API layer and
the disk interface. Following diagram illustrates the layer separation:

+--------------------------+
| Zephyr File System API |
+--------------------------+
| Glue layer for FS module |
+--------------------------+
| File System module |
+--------------------------+
| Disk I/O interface |
+--------------------------+
| Storage media interface |
+--------------------------+

The File system module would be an implementation that provides clean
separation between the API layer and disk I/O. A glue layer will
abstract its own interface to that of the Zephyr FS API. The disk i/o
interface will use Zephyr's flash HAL.
Have you thought about possibly consolidating the API, the glue layer
and the filesystem itself when there is only one filesystem in the
system, which might well be the most common case ? Basically having the
ZFS API become a set of macros that map directly to the interface of the
one File System module present in the system ?


Thomas, Ramesh
 

On 08/05/2016 10:21 AM, Benjamin Walsh wrote:
On Tue, Jul 26, 2016 at 06:36:45PM -0700, Ramesh Thomas wrote:
A file system is necessary to store data reliably and to be accessed in
a consistent and deterministic manner. Example applications are data
logging, media files and file based databases which would need to be
stored for later processing off-line or to be sent over radio in bursts
instead of per sample captured.

Related requirement:
https://jira.zephyrproject.org/browse/ZEP-228

***Proposal***
The Zephyr File System API will follow interfaces similar to stdio and
POSIX so it is familiar to users. However, it does not claim to be
compliant to POSIX or any other specification.

One of the goals is to keep clean separation of the file API layer and
the disk interface. Following diagram illustrates the layer separation:

+--------------------------+
| Zephyr File System API |
+--------------------------+
| Glue layer for FS module |
+--------------------------+
| File System module |
+--------------------------+
| Disk I/O interface |
+--------------------------+
| Storage media interface |
+--------------------------+

The File system module would be an implementation that provides clean
separation between the API layer and disk I/O. A glue layer will
abstract its own interface to that of the Zephyr FS API. The disk i/o
interface will use Zephyr's flash HAL.
Have you thought about possibly consolidating the API, the glue layer
and the filesystem itself when there is only one filesystem in the
system, which might well be the most common case ? Basically having the
ZFS API become a set of macros that map directly to the interface of the
one File System module present in the system ?
Yes, I agree that we will most likely have only one file system. I am
looking into making the glue layer as thin as possible.

Some functions have different parameters, while almost all of them need
to translate the error value that is returned by the file system module.
The ZFS API returns -ERRNO values and one of its goals is to provide a
generic API familiar to users. The glue layer does this translation.