The MRC CLIMB BIG DATA project offers modern storage in the form of S3 buckets, an object storage introduced by Amazon Web Services (AWS).

The problem

Traditional volumes behaves like local hard disks, while AWS S3 buckets are usually operated in a different way. Here we will see how to mount an AWS S3 bucket (provided by CLIMB BIG DATA) into a Linux machine (typically, a virtual machine).

What to know

From the Bryn interface take note of:

• Your bucket name (2), something like group-storage
• Your Access Key (4), something like OYXIA829VSHI3X9WE
• Your Secret (5), something like Q6Pr0chIsHereQ8rCR1B10KZa0s7kujiaKKu
• Where to mount the bucket (an empty directory, which we will create from scratch in this tutorial)

The solution

1) Install fuse

1
2

sudo apt update
sudo apt install -y fuse

2) Install rclone (you can download the binary from here)

1

sudo -v ; curl https://rclone.org/install.sh | sudo bash

3) Configure rclone. Find where the configuration file is located with:

1

rclone config file

The output should be something like /home/ubuntu/.config/rclone/rclone.conf

4) Using an editor - we will use nano - to edit/create the file. The command to start the editing session is:

1
2

# Replace the filename with the output of the previous point!!!
nano /home/ubuntu/.config/rclone/rclone.conf

in the configuration file paste the following, changing the informations with your bucket name, access ID and secret (see What you need to know):

1
2
3
4
5
6
7

[climb]
type = s3
provider = Ceph
access_key_id = <your-CLIMB-S3-access-key>
secret_access_key = <your-CLIMB-S3-secret-access-key>
endpoint = https://s3.climb.ac.uk
acl = private

Use the Ctrl + O keystroke to save the file, then Ctrl + X to exit nano.

5) Create a directory where to mount the bucket. For example /group-data:

1
2
3

# Replace "/group-data" with the desired location
sudo mkdir /group-data
sudo chown -R $USER:$GROUP  /group-data

if you create a directory in your home, for example ~/bucket, you won’t need sudo.

6) Mount the bucket to the newly created directory:

1
2
3

# Replace "BUCKET-NAME" with your bucket name (2)
# Replace "/group-data" with the desired mounting point
rclone mount --daemon climb:BUCKET-NAME /group-data