Discussion:
Can anyone confirm or deny if ecryptfs will work with a glusterfs backend?
Lance Reed
2014-02-26 16:06:20 UTC
Permalink
I am attempting to setup encrypted user home directories via eCryptfs using
gluster as a backend.

Very simple setup currently has a small two node gluster cluster mounted by
a separate client. Normal gluster client and NFS mount / file options are
working fine.


e.g. https://wiki.archlinux.org/index.php/ECryptfs#Encrypting_a_home_directory

In my attempts lay ecryptfs on top of the mounted native gluster setup, I am
unable to edit a file, write etc. I either get zero length or fixed sizes.

Only log messages I get are:
"Either the lower file is not in a valid eCryptfs format, or the key could
not be retrieved. Plaintext passthrough mode is not enabled; returning -EIO"

I am posting in this forum to see if anyone knows of any reason why this may
be failing from the ecryptfs side and I should stop banging my head against
the wall...

I am trying Centos / RHEL.
See these bugs:
Bug 762976 - (GLUSTER-1244) ecryptfs does not work when the directory to be
encrypted is on gluster mount
https://bugzilla.redhat.com/show_bug.cgi?id=762976

A non-empty file created on glusterfs with ecryptfs reports as a file of
size zero
https://bugzilla.redhat.com/show_bug.cgi?id=989702#c1

These look to be issues with O_DIRECT usage in fuse but are suppose to be
fixed now.

I was hoping someone might have an idea or remember some of this to help me
figure out if using glusterfs for a backend with eCryptfs is even an option.

Is it possible that this bug is still the core problem?
"ecryptfs does not work properly over nfs, cifs, samba, WebDAV, or aufs"
https://bugs.launchpad.net/ecryptfs/+bug/277578
It is old but still seems to be open..

versions of the code I am using:
glusterfs-cli-3.4.2-1.el6.x86_64
glusterfs-libs-3.4.2-1.el6.x86_64
glusterfs-fuse-3.4.2-1.el6.x86_64
glusterfs-server-3.4.2-1.el6.x86_64
ecryptfs-utils-82-6.el6_1.3.x86_64
glusterfs-3.4.2-1.el6.x86_64


Thanks very much in advance for any ideas on the problem.
Tyler Hicks
2014-02-26 18:00:45 UTC
Permalink
Hi Lance!
Post by Lance Reed
I am attempting to setup encrypted user home directories via eCryptfs using
gluster as a backend.
Very simple setup currently has a small two node gluster cluster mounted by
a separate client. Normal gluster client and NFS mount / file options are
working fine.
e.g. https://wiki.archlinux.org/index.php/ECryptfs#Encrypting_a_home_directory
In my attempts lay ecryptfs on top of the mounted native gluster setup, I am
unable to edit a file, write etc. I either get zero length or fixed sizes.
"Either the lower file is not in a valid eCryptfs format, or the key could
not be retrieved. Plaintext passthrough mode is not enabled; returning -EIO"
I am posting in this forum to see if anyone knows of any reason why this may
be failing from the ecryptfs side and I should stop banging my head against
the wall...
I am trying Centos / RHEL.
Bug 762976 - (GLUSTER-1244) ecryptfs does not work when the directory to be
encrypted is on gluster mount
https://bugzilla.redhat.com/show_bug.cgi?id=762976
A non-empty file created on glusterfs with ecryptfs reports as a file of
size zero
https://bugzilla.redhat.com/show_bug.cgi?id=989702#c1
These look to be issues with O_DIRECT usage in fuse but are suppose to be
fixed now.
I was hoping someone might have an idea or remember some of this to help me
figure out if using glusterfs for a backend with eCryptfs is even an option.
eCryptfs mounted on top of glusterfs is something that I've never tried
and I don't recall anyone talking with upstream eCryptfs about it,
either. It wouldn't surprise me if it doesn't work. :/

I haven't paid much attention to glusterfs, but I thought the answer to
encryption with glusterfs was hekafs?

While briefly refreshing my memory on hekafs, it sounds like it is
geared towards cloud storage providers. Maybe it is too complex for your
needs?
Post by Lance Reed
Is it possible that this bug is still the core problem?
"ecryptfs does not work properly over nfs, cifs, samba, WebDAV, or aufs"
https://bugs.launchpad.net/ecryptfs/+bug/277578
It is old but still seems to be open..
That bug is a mess. It needs to be reevaluated and split up into
separate bug reports for individual lower filesystems. There is no
single fix for that bug and it will never be closed in its current
state.
Post by Lance Reed
glusterfs-cli-3.4.2-1.el6.x86_64
glusterfs-libs-3.4.2-1.el6.x86_64
glusterfs-fuse-3.4.2-1.el6.x86_64
glusterfs-server-3.4.2-1.el6.x86_64
ecryptfs-utils-82-6.el6_1.3.x86_64
glusterfs-3.4.2-1.el6.x86_64
Kernel version?

Tyler
Lance Reed
2014-02-26 18:35:09 UTC
Permalink
Tyler,

Thanks for the response!
Red Hat Enterprise Linux Server release 6.4 (Santiago) (and Centos 6.4)
2.6.32-358.14.1.el6.x86_64

I am actually planning to use this in a cloud setup. Mostly for an internal
shared NFS like but HA space.
I had thought HekaFS had gone a bit dormant...(not a lot of talk since May
2013...)
I will look at it again. Thanks for the tip and taking a look.
Any thought would be greatly appreciated.

Cheers,

Lance


If this is a double post, I apologize.. oddness on email vs web interface:
Tyler Hicks
2014-02-26 19:56:51 UTC
Permalink
Post by Lance Reed
Tyler,
Thanks for the response!
Red Hat Enterprise Linux Server release 6.4 (Santiago) (and Centos 6.4)
2.6.32-358.14.1.el6.x86_64
While I've not made any recent changes that I would expect to improve
glusterfs support, that is a somewhat old kernel and a newer version
may, obviously, have different results.

However, I should say that even if eCryptfs suddenly works on top of
glusterfs in a newer kernel, there's still a fundamental problem with
eCryptfs stacked on top of any type of remote filesystem. eCryptfs does
not expect the lower/backend filesystem to be modified by another
machine. So you could have cache coherency problems if two or more
clients are reading a file and then one of them updates the file.

Fixing the cache coherency issues would be step 2, after eCryptfs
actually starts working on top of popular remote filesystems (nfs, cifs,
glusterfs, etc.). Unfortunately, I'm not aware of anyone working on step
1 at the moment.

Tyler
Post by Lance Reed
I am actually planning to use this in a cloud setup. Mostly for an internal
shared NFS like but HA space.
I had thought HekaFS had gone a bit dormant...(not a lot of talk since May
2013...)
I will look at it again. Thanks for the tip and taking a look.
Any thought would be greatly appreciated.
Cheers,
Lance
--
To unsubscribe from this list: send the line "unsubscribe ecryptfs" in
More majordomo info at http://vger.kernel.org/majordomo-info.html
Loading...