How to Mount S3 Bucket on CentOS and Ubuntu using S3FS
Posted on February 1st, 2017
In this tutorial we can check how to mount S3 bucket on your InterServer VPS or Dedicated Server.
S3FS is a FUSE (File System in User Space) will mount Amazon S3 as a local file system. S3FS has an ability to manipulate Amazon S3 bucket in many useful ways. If you wish to access your Amazon S3 bucket without mounting it on your server, you can use s3cmd command line utility to manage S3 bucket.
What is an Amazon S3 bucket?
Amazon S3 is a cloud based web service interface that you can used to store and retrieve any amount of data. To upload your data, first you need to create an S3 bucket in one of the Amazon regions.
Creating a Bucket
S3 provides an API for creating and managing buckets. You can create a maximum of 100 buckets from your AWS console. When you create a bucket, you need to provide a name and AWS region where you want to create the bucket. In each bucket, you can store any number of objects. You can use your AWS account root credentials to create a bucket, but it is not recommended. Instead just create an IAM user and add full permission to that user on S3 bucket. You can access your S3 bucket from your Amazon S3 console.
Please follow the below steps to mount s3 bucket on your server.
1) Remove Existing Packages
Before installing any package, first you need to check if you have any existing fuse or S3FS on your server. If it is already existing, then remove it from your server to avoid further conflicts. Use the following command to check if you have any existing fuse or S3FS on your server
CentOS users:
$ yum remove fuse fuse-s3fs
Ubuntu Users:
$ apt-get remove fuse
2) Install Packages
Install all dependency packages for fuse and s3cmd using the below command.
CentOS users:
$ yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap
Ubuntu Users:
$ apt-get install build-essential libcurl4-openssl-dev libxml2-dev mime-support
3) Download and Compile Fuse
Move to /usr/src then download and compile fuse source code. After compiling, add fuse to kernel. In my case the latest version of fuse is fuse-3.0.0
$ cd /usr/src/
$ wget https://github.com/libfuse/libfuse/releases/download/fuse-3.0.0/fuse-3.0.0.tar.gz
$ tar xzf fuse-3.0.0.tar.gz
$ cd fuse-3.0.0
$ ./configure –prefix=/usr/local
$ make && make install
$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
$ ldconfig
$ modprobe fuse
4) Download and compile S3FS
Navigate to /usr/src, Download and compile s3fs source code.
$ git clone https://github.com/s3fs-fuse/s3fs-fuse.git
$ cd s3fs-fuse
$ ./autogen.sh
$ ./configure
$ make
$ make install
5) Setup Access Key
Both access key and secret key of your s3 AWS account is required for configuring S3FS. Replace the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your actual key values.
$ vi /etc/passwd-s3fs
AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY
Make sure that the file has proper permission.
$ chmod 600 /etc/passwd-s3fs
6) Mount S3 Bucket
You can run the below command to mount s3fs.
$ s3fs mybucket /path/to/mountpoint -o passwd_file=/etc/passwd-s3fs
You can also mount the s3 bucket on boot by following below commands.
$ mkdir /tmp/cache
$ mkdir /path/to/mountpoint
$ chmod 777 /tmp/cache /path/to/mountpoint
$ vi /etc/fstab
s3fs#<mybucket> /path/to/mountpoint fuse allow_other,use_cache=/tmp/cache,uid=userid,gid=groupid 0 0
$ mount -a
Congratulations you have successfully mounted s3 bucket on your server.
If you need any further assistance please contact our support department.
s3fs `./configure` fails every time on CentOS 7 with:
…
configure: error: Package requirements (fuse >= 2.8.4 libcurl >= 7.0 libxml-2.0 >= 2.6) were not met:
No package ‘fuse’ found
Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.
Alternatively, you may set the environment variables common_lib_checking_CFLAGS
and common_lib_checking_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
You are getting that error because package requirement is not met. The required packages and the required versions are mentioned in the error report and you have to install the right versions to make this work. Since this is considered as third-party our support team may not be able to do this for you. But if you have tried everything and if nothing is working, we can take a look : https://www.interserver.net/resolve
See below.
If you already installed fuse, try to install “fuse-devel”.
#> yum install fuse-devel -y
and run it again.
This is the way I fixed on my CentOS 7, For your reference.
@Jithin & @Johnny please note this ticket for details on the failure:
https://github.com/s3fs-fuse/s3fs-fuse/issues/536
Does anyone know how to make this accessible by ALL users on the linux Server?
Right not when I run the s3fs command as root.
s3fs #BUCKETNAME /mnt/AWSS3 -o passwd_file=/etc/passwd-s3fs -o allow_other,use_cache=/root/cache
OR
s3fs bbcorp-qa /mnt/AWSS3 -o passwd_file=/etc/passwd-s3fs -o allow_other, uid=1000, umask=077, gid=1000, use_cache=/root/cache
The full directory structure is accessible by the root user ONLY, any other user that trys to access a file or directory get the following error:
ls -al /mnt/AWSS3/user
ls: cannot open directory /mnt/AWSS3/user: Operation not permitted
cat /mnt/AWSS3/mytest.txt
cat: /mnt/AWSS3/mytest.txt: Permission denied
Is there a switch that can help me with the access to ALL users or a specific one?
Actually -o uid=1000,umask=077,gid=1000,… will do it. Make sure you provide correct uid and gid of the user that you want to allow access. ( Run “id -u username” to get the uid and “id -g username” to get uid. Replace “username” with the appropriate user’s username )
Hi,
Unfortunately, it doesn’t work for me
# mkdir /tmp/s3
# id alex
uid=1000(alex) gid=1001(alex) groups=1001(alex)
# s3fs my-bucket /tmp/s3 -o passwd_file=/etc/passwd-s3fs,uid=1000,gid=1001,umask=077
# su – alex
$ ls -las /tmp/s3
ls: cannot access ‘/tmp/s3’: Permission denied
# la -ld /tmp/s3
drwx—— 1 alex alex 0 Jan 1 1970 /tmp/s3
#mount
s3fs on /tmp/s3 type fuse.s3fs (rw,nosuid,nodev,relatime,user_id=0,group_id=0)
Please advise
Try full permission for the mount point.
Hi there
I have tried to give full permission to the mount point with Chmod 777 -R /mnt/uploadS3, but I get also the “Input/output error”.
My problem is that I am able to write and copy files in the mount point with the “root” user, but not for example with other users like “tomcat”.
Any advice??
Unable to create files: I have compiled and installed libfuse 2.9.7 and latest build of S3fs on centos 6.9. I am able to mount S3 bucket a.b.com on /home/backup. No errors reported in logs. But I am unable to create any folder on my Linux server. Also folders in bucket visible on S3 console is not present on my linux server mount point folder. I am also unable to create new folder or file on linux server mounted folder.
Command used is s3fs a.b.com /home/backup -o passwd_file=/root/.passwd-s3fs,curldbg -o use_path_request_style -o allow_other
Error is: mkdir: cannot create directory `file’: File exists
Check the mount point permission.
Mounting the my S3 bucket via the cli “s3fs mybucket /path/to/mountpoint -o passwd_file=/etc/passwd-s3fs” works fine
but putting the following statement in the /etc/fstab:
s3fs#s3-ftp-energiscloud /mnt/s3mnt fuse allow_other,use_cache=/tmp/cache,uid=500,gid=500 0 0
gives the following error when executing mount:
mount: wrong fs type, bad option, bad superblock on s3fs#s3-ftp-energiscloud,
missing codepage or helper program, or other error
In some cases useful info is found in syslog – try
dmesg | tail or so.
Can someone help me ? thanks
I’m having the same issue, the syntax for fstab doesn’t work for me or is missing a lot of parameters like the password file, API URL, etc. or is none of that needed for fstab?
Here are the commands I used on Amazon Linux:
cd /usr/src/
wget https://github.com/libfuse/libfuse/releases/download/fuse-3.0.0/fuse-3.0.0.tar.gz
tar xzf fuse-3.0.0.tar.gz
cd fuse-3.0.0
./configure –prefix=/usr/local
make && make install
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
ldconfig
modprobe fuse
cd ..
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse/
yum install automake
./autogen.sh
./configure
yum install fuse-devel
./configure
make
make install
Thanks for the great walkthrough!
The dependencies for s3fs have changed a little in the meantime: https://github.com/s3fs-fuse/s3fs-fuse
For RHEL/CentOS:
sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
For Ubuntu:
sudo apt-get install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
s3fs is available via yum now, by the way.
yum search s3fs
yum install s3fs-fuse.x86_64
In my ubuntu instance using s3sf mounting s3 bucket successfully, but when I am getting objects it through 403 forbidden error. I am trying through command for permission denied, but it not working. How to solve 403 error.