Hi,

I was trying to implement fs_truncate() for NFFS but this does not
seem to be possible in a reasonable way, at least not with current
NFFS API. The only option I see now is to copy contents to new file
and unlink old one, but this has obvious disadvantages: it's slow, we
may easily run out of space etc.

So do we really need option to truncate any opened file to any size?
In Mynewt there's only option to truncate file to 0 when opening it by
specifying flag to fs_open() and this seems to be just enough. There
seems to be not many references to fs API right now (actually
Bluetooth subsystem is the only one) so perhaps we could just remove
fs_truncate() and allow truncate to 0 using something else?

My proposal would be one of:
- add flag to fs_open()
- make fs_truncate() truncate file specified by *path* to 0

The reason why I'd prefer to have truncation done by *path* instead of
*fp* is because it's simpler to implement in NFFS since what needs to
be done is to unlink old file and create new one. The API to create
new file takes full path as a parameter so having fp only makes it a
bit more complicated.

Best regards,
Andrzej

Hello,

On Mon, 4 Sep 2017 09:27:30 +0200
Andrzej Kaczmarek <andrzej.kaczmarek@...> wrote:

Hi,

I was trying to implement fs_truncate() for NFFS but this does not
seem to be possible in a reasonable way, at least not with current
NFFS API. The only option I see now is to copy contents to new file
and unlink old one, but this has obvious disadvantages: it's slow, we
may easily run out of space etc.

ENOTSUP is your friend (except Linux thinks it should be EPERM, see
below).

So do we really need option to truncate any opened file to any size?

POSIX has it:
http://pubs.opengroup.org/onlinepubs/7908799/xsh/ftruncate.html , so we
apparently need it too.

In Mynewt there's only option to truncate file to 0 when opening it by
specifying flag to fs_open() and this seems to be just enough.There
seems to be not many references to fs API right now (actually
Bluetooth subsystem is the only one) so perhaps we could just remove
fs_truncate() and allow truncate to 0 using something else?

In a Linux system, only 0.0001% or so of the source code belongs to the
kernel. With that ratio in mind, how many Zephyr applications have you
considered?

My proposal would be one of:
- add flag to fs_open()
- make fs_truncate() truncate file specified by *path* to 0

Let's have a look at https://linux.die.net/man/2/ftruncate :

truncate(): /* Since glibc 2.12: */ _POSIX_C_SOURCE >= 200809L
ftruncate(): /* Since glibc 2.3.5: */ _POSIX_C_SOURCE >= 200112L

So, ftruncate() was categorized in glibc since the break of the century,
while truncate()-by-path is much more of a novelty, which should set
a priority of the order of their support.

The reason why I'd prefer to have truncation done by *path* instead of
*fp* is because it's simpler to implement in NFFS since what needs to
be done is to unlink old file and create new one. The API to create
new file takes full path as a parameter so having fp only makes it a
bit more complicated.

This seems like a logical fallacy: you have a problem with a single
thing on your hands (NFFS) and you want to change the world around it
(Zephyr API). Just accepting that NFFS is a novel, unproven,
underdeveloped and undertested filesystem should give enough of a
solution: just treat ftruncate operation as unsupported on it. Another
solution of course is to fix NFFS to support a standard POSIX
filesystem operation repertoire.

Let's look at https://linux.die.net/man/2/ftruncate again:

EPERM
The underlying file system does not support extending a file beyond its
current size.

However, my local man has more of it:

EPERM The underlying filesystem does not support extending
a file beyond its current size.

EPERM The operation was prevented by a file seal; see fcntl(2).

So, in Linux, EPERM is ambiguous. There's actually a better error code
- ENOTSUP. Arguably, it's the same "POSIX drama" as with EPERM vs
EACCESS, vividly explained here:
http://blog.unclesniper.org/archives/2-Linux-programmers,-learn-the-difference-between-EACCES-and-EPERM-already!.html
Following the same logic, I'd recommend to use ENOTSUP here (then wait
until Zephyr's POSIX subsystem matures, apps getting ported, and break
because of the "non-Linuxy" error code is used, then we switch it to
EPERM).

Best regards,
Andrzej

--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

On Mon, Sep 04, 2017 at 09:27:30AM +0200, Andrzej Kaczmarek wrote:

I was trying to implement fs_truncate() for NFFS but this does not
seem to be possible in a reasonable way, at least not with current
NFFS API. The only option I see now is to copy contents to new file
and unlink old one, but this has obvious disadvantages: it's slow, we
may easily run out of space etc.

If your goal is posix compliance, this isn't going to be right anyway.
truncate is required to be atomic.

The POSIX truncate call is also allowed to be used to extend the size
of a file.

So do we really need option to truncate any opened file to any size?
In Mynewt there's only option to truncate file to 0 when opening it by
specifying flag to fs_open() and this seems to be just enough. There
seems to be not many references to fs API right now (actually
Bluetooth subsystem is the only one) so perhaps we could just remove
fs_truncate() and allow truncate to 0 using something else?

My proposal would be one of:
- add flag to fs_open()
- make fs_truncate() truncate file specified by *path* to 0

The reason why I'd prefer to have truncation done by *path* instead of
*fp* is because it's simpler to implement in NFFS since what needs to
be done is to unlink old file and create new one. The API to create
new file takes full path as a parameter so having fp only makes it a
bit more complicated.

I think it would be better to return ENOTSUP than to implement it in a
non-compliant way. My suggestion would be to document it as not
supported, and make a ticket to add support to NFFS for truncation
later.

David

On Tue, Sep 05, 2017 at 10:18:01AM +0200, Andrzej Kaczmarek wrote:

I'm not really sure if this needs to be POSIX compliant - the Jira ticket for
adding filesystem APIs states that it is designed after POSIX but is not POSIX
compliant. For example, fs_open() does not have flags which could be used to
truncate existing file when opening, so it would cover one of possible
scenarios here. But perhaps something changed since then, don't know.

Perhaps we should change the API to add the truncate option to
fs_open(), at least provided at least one underlying implementation
supports this.

Yes, currently I return ENOTSUP for this. My only concern here is that this
means there is no way to open existing file and truncate it other than
fs_unlink() + fs_open() due to missing flags in fs_open() I mentioned above.
BTW, does it make sense to have fs_truncate() work for '0' as offset and return
e.g. EINVAL for other values? This should be doable I think.

Only if the fs_truncate() for 0 offset can be implemented atomically.
I'm not sure how that would work for operations given by file
descriptor, since the file is already opened.

I think there should be as little between the Zephyr fs API and the
NFFS API. A developer using a filesystem on an embedded system needs
to understand what is happening (and specifically what will happen if
power is interrupted during an operation). That means we should avoid
having API calls that try to simulate behavior that isn't there.

For example, if fs_truncate() somehow were to use fs_open() by
figuring out the filename the existing file descriptor used, and
opening with truncation in NFFS, this would cause an additional file
descriptor to be used for the operation (even if just momentarily).
In a system design, it is possible that the developer will configure
the exact number of file descriptors available, which could cause this
operation to either mysteriously fail, or succeed, but use the file
descriptor temporarily, causing another thread's filesystem operation
to fail.

So, my suggestion would be to add the O_TRUNC (or equivalent) flag to
open, make fs_truncate() return ENOTSUP, and just document this. If
arbitrary truncation, or truncation of an open file is desired, that
can be added later to NFFS, and then to the API.

David