centos7使用ossfs挂载阿里云oss的bucket到本地文件系统

发布于 2017-09-13  2.88k 次阅读


简介

ossfs 可以让 Linux/Mac OS X 系统把Aliyun OSS bucket 挂载到本地文件系统中,能够便捷的通过本地文件系统操作OSS上的对象,实现数据的共享。

功能

ossfs 基于s3fs 构建,具有s3fs 的全部功能。主要功能包括:

  • 支持POSIX 文件系统的大部分功能,包括文件读写,目录,链接操作,权限, uid/gid,以及扩展属性(extended attributes)
  • 通过OSS 的multipart 功能上传大文件。
  • MD5 校验保证数据完整性。

安装ossfs

[root@tokyo ~]# yum install automake gcc-c++ git libcurl-devel libxml2-devel fuse-devel make openssl-devel
Loaded plugins: fastestmirror
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
base                                                       | 3.6 kB  00:00:00     
elrepo-kernel                                              | 2.9 kB  00:00:00     
epel/x86_64/metalink                                       | 5.4 kB  00:00:00     
epel                                                       | 4.3 kB  00:00:00     
extras                                                     | 3.4 kB  00:00:00     
updates                                                    | 3.4 kB  00:00:00     
(1/6): epel/x86_64/group_gz                                | 170 kB  00:00:00     
(2/6): epel/x86_64/updateinfo                              | 821 kB  00:00:00     
(3/6): extras/7/x86_64/primary_db                          | 191 kB  00:00:00     
(4/6): epel/x86_64/primary_db                              | 4.8 MB  00:00:00     
(5/6): elrepo-kernel/x86_64/primary_db                     | 1.7 MB  00:00:01     
(6/6): updates/7/x86_64/primary_db                         | 7.8 MB  00:00:01     
Determining fastest mirrors
 * base: ftp.riken.jp
 * epel: ftp.riken.jp
 * extras: ftp.riken.jp
 * updates: ftp.riken.jp
Package automake-1.13.4-3.el7.noarch already installed and latest version
Package gcc-c++-4.8.5-11.el7.x86_64 already installed and latest version
Package git-1.8.3.1-6.el7_2.1.x86_64 already installed and latest version
Package libcurl-devel-7.29.0-35.el7.centos.x86_64 already installed and latest version
Package libxml2-devel-2.9.1-6.el7_2.3.x86_64 already installed and latest version
Package 1:make-3.82-23.el7.x86_64 already installed and latest version
Package 1:openssl-devel-1.0.1e-60.el7_3.1.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package fuse-devel.x86_64 0:2.9.2-7.el7 will be installed
--> Processing Dependency: fuse-libs = 2.9.2-7.el7 for package: fuse-devel-2.9.2-7.el7.x86_64
--> Processing Dependency: libulockmgr.so.1()(64bit) for package: fuse-devel-2.9.2-7.el7.x86_64
--> Processing Dependency: libfuse.so.2()(64bit) for package: fuse-devel-2.9.2-7.el7.x86_64
--> Running transaction check
---> Package fuse-libs.x86_64 0:2.9.2-7.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================
 Package              Arch             Version               Repository      Size
==================================================================================
Installing:
 fuse-devel           x86_64           2.9.2-7.el7           base            37 k
Installing for dependencies:
 fuse-libs            x86_64           2.9.2-7.el7           base            93 k

Transaction Summary
==================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 129 k
Installed size: 415 k
Is this ok [y/d/N]: y
Downloading packages:
(1/2): fuse-libs-2.9.2-7.el7.x86_64.rpm                    |  93 kB  00:00:00     
(2/2): fuse-devel-2.9.2-7.el7.x86_64.rpm                   |  37 kB  00:00:00     
----------------------------------------------------------------------------------
Total                                                274 kB/s | 129 kB  00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : fuse-libs-2.9.2-7.el7.x86_64                                   1/2 
  Installing : fuse-devel-2.9.2-7.el7.x86_64                                  2/2 
  Verifying  : fuse-devel-2.9.2-7.el7.x86_64                                  1/2 
  Verifying  : fuse-libs-2.9.2-7.el7.x86_64                                   2/2 

Installed:
  fuse-devel.x86_64 0:2.9.2-7.el7                                                 

Dependency Installed:
  fuse-libs.x86_64 0:2.9.2-7.el7                                                  

Complete!

[root@tokyo ~]# wget https://github.com/aliyun/ossfs/archive/v1.80.2.tar.gz
--2017-09-13 16:39:35--  https://github.com/aliyun/ossfs/archive/v1.80.2.tar.gz
Resolving github.com (github.com)… 192.30.255.113, 192.30.255.112
Connecting to github.com (github.com)|192.30.255.113|:443… connected.
HTTP request sent, awaiting response… 302 Found
Location: https://codeload.github.com/aliyun/ossfs/tar.gz/v1.80.2 [following]
--2017-09-13 16:39:35--  https://codeload.github.com/aliyun/ossfs/tar.gz/v1.80.2
Resolving codeload.github.com (codeload.github.com)… 192.30.253.120, 192.30.253.121
Connecting to codeload.github.com (codeload.github.com)|192.30.253.120|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘v1.80.2.tar.gz’

    [  <=>                                    ] 134,056      246KB/s   in 0.5s   

2017-09-13 16:39:36 (246 KB/s) - ‘v1.80.2.tar.gz’ saved [134056]


[root@tokyo ~]# tar xvf v1.80.2.tar.gz 
ossfs-1.80.2/
ossfs-1.80.2/.gitignore
ossfs-1.80.2/.mailmap
ossfs-1.80.2/.travis.yml
ossfs-1.80.2/AUTHORS
ossfs-1.80.2/COPYING
ossfs-1.80.2/ChangeLog
ossfs-1.80.2/INSTALL
ossfs-1.80.2/Makefile.am
ossfs-1.80.2/README.md
ossfs-1.80.2/autogen.sh
ossfs-1.80.2/configure.ac
ossfs-1.80.2/doc/
ossfs-1.80.2/doc/Makefile.am
ossfs-1.80.2/doc/man/
ossfs-1.80.2/doc/man/ossfs.1
ossfs-1.80.2/scripts/
ossfs-1.80.2/scripts/automount.template
ossfs-1.80.2/scripts/build-pkg.py
ossfs-1.80.2/src/
ossfs-1.80.2/src/Makefile.am
ossfs-1.80.2/src/cache.cpp
ossfs-1.80.2/src/cache.h
ossfs-1.80.2/src/common.h
ossfs-1.80.2/src/common_auth.cpp
ossfs-1.80.2/src/curl.cpp
ossfs-1.80.2/src/curl.h
ossfs-1.80.2/src/fdcache.cpp
ossfs-1.80.2/src/fdcache.h
ossfs-1.80.2/src/gnutls_auth.cpp
ossfs-1.80.2/src/nss_auth.cpp
ossfs-1.80.2/src/openssl_auth.cpp
ossfs-1.80.2/src/s3fs.cpp
ossfs-1.80.2/src/s3fs.h
ossfs-1.80.2/src/s3fs_auth.h
ossfs-1.80.2/src/s3fs_util.cpp
ossfs-1.80.2/src/s3fs_util.h
ossfs-1.80.2/src/string_util.cpp
ossfs-1.80.2/src/string_util.h
ossfs-1.80.2/src/test_string_util.cpp
ossfs-1.80.2/src/test_util.h
ossfs-1.80.2/test/
ossfs-1.80.2/test/Makefile.am
ossfs-1.80.2/test/integration-test-common.sh
ossfs-1.80.2/test/integration-test-main.sh
ossfs-1.80.2/test/mergedir.sh
ossfs-1.80.2/test/passwd-s3fs
ossfs-1.80.2/test/require-root.sh
ossfs-1.80.2/test/s3proxy.conf
ossfs-1.80.2/test/sample_ahbe.conf
ossfs-1.80.2/test/sample_delcache.sh
ossfs-1.80.2/test/small-integration-test.sh
ossfs-1.80.2/test/stat_cache_test.py

[root@tokyo ~]# cd ossfs-1.80.2/

[root@tokyo ossfs-1.80.2]# ./autogen.sh 
--- Make commit hash file -------
--- Finished commit hash file ---
--- Start autotools -------------
configure.ac:27: installing './config.guess'
configure.ac:27: installing './config.sub'
configure.ac:28: installing './install-sh'
configure.ac:28: installing './missing'
src/Makefile.am: installing './depcomp'
parallel-tests: installing './test-driver'
--- Finished autotools ----------

[root@tokyo ossfs-1.80.2]# ./configure 
checking build system type… x86_64-unknown-linux-gnu
checking host system type… x86_64-unknown-linux-gnu
checking target system type… x86_64-unknown-linux-gnu
checking for a BSD-compatible install… /usr/bin/install -c
checking whether build environment is sane… yes
checking for a thread-safe mkdir -p… /usr/bin/mkdir -p
checking for gawk… gawk
checking whether make sets $(MAKE)… yes
checking whether make supports nested variables… yes
checking for g++… g++
checking whether the C++ compiler works… yes
checking for C++ compiler default output file name… a.out
checking for suffix of executables… 
checking whether we are cross compiling… no
checking for suffix of object files… o
checking whether we are using the GNU C++ compiler… yes
checking whether g++ accepts -g… yes
checking for style of include used by make… GNU
checking dependency style of g++… gcc3
checking for gcc… gcc
checking whether we are using the GNU C compiler… yes
checking whether gcc accepts -g… yes
checking for gcc option to accept ISO C89… none needed
checking dependency style of gcc… gcc3
checking ossfs build with nettle(GnuTLS)… no
checking ossfs build with OpenSSL… no
checking ossfs build with GnuTLS… no
checking ossfs build with NSS… no
checking for pkg-config… /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0… yes
checking for common_lib_checking… yes
checking compile ossfs with… OpenSSL
checking for DEPS… yes
checking for malloc_trim… yes
checking pthread mutex recursive… PTHREAD_MUTEX_RECURSIVE
checking for git… yes
checking for .git… no
checking github short commit hash… unknown
checking that generated files are newer than configure… done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating test/Makefile
config.status: creating doc/Makefile
config.status: creating config.h
config.status: executing depfiles commands

[root@tokyo ossfs-1.80.2]# make && make install
make  all-recursive
make[1]: Entering directory `/root/ossfs-1.80.2'
Making all in src
make[2]: Entering directory `/root/ossfs-1.80.2/src'
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2      -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT s3fs.o -MD -MP -MF .deps/s3fs.Tpo -c -o s3fs.o s3fs.cpp
mv -f .deps/s3fs.Tpo .deps/s3fs.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2      -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT curl.o -MD -MP -MF .deps/curl.Tpo -c -o curl.o curl.cpp
mv -f .deps/curl.Tpo .deps/curl.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2      -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT cache.o -MD -MP -MF .deps/cache.Tpo -c -o cache.o cache.cpp
mv -f .deps/cache.Tpo .deps/cache.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2      -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT string_util.o -MD -MP -MF .deps/string_util.Tpo -c -o string_util.o string_util.cpp
mv -f .deps/string_util.Tpo .deps/string_util.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2      -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT s3fs_util.o -MD -MP -MF .deps/s3fs_util.Tpo -c -o s3fs_util.o s3fs_util.cpp
mv -f .deps/s3fs_util.Tpo .deps/s3fs_util.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2      -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT fdcache.o -MD -MP -MF .deps/fdcache.Tpo -c -o fdcache.o fdcache.cpp
mv -f .deps/fdcache.Tpo .deps/fdcache.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2      -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT common_auth.o -MD -MP -MF .deps/common_auth.Tpo -c -o common_auth.o common_auth.cpp
mv -f .deps/common_auth.Tpo .deps/common_auth.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2      -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT openssl_auth.o -MD -MP -MF .deps/openssl_auth.Tpo -c -o openssl_auth.o openssl_auth.cpp
mv -f .deps/openssl_auth.Tpo .deps/openssl_auth.Po
g++  -g -O2 -Wall -D_FILE_OFFSET_BITS=64   -o ossfs s3fs.o curl.o cache.o string_util.o s3fs_util.o fdcache.o common_auth.o openssl_auth.o   -pthread -lfuse -lcurl -lxml2 -lcrypto   
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2      -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT test_string_util.o -MD -MP -MF .deps/test_string_util.Tpo -c -o test_string_util.o test_string_util.cpp
mv -f .deps/test_string_util.Tpo .deps/test_string_util.Po
g++  -g -O2 -Wall -D_FILE_OFFSET_BITS=64   -o test_string_util string_util.o test_string_util.o  
make[2]: Leaving directory `/root/ossfs-1.80.2/src'
Making all in test
make[2]: Entering directory `/root/ossfs-1.80.2/test'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/ossfs-1.80.2/test'
Making all in doc
make[2]: Entering directory `/root/ossfs-1.80.2/doc'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/ossfs-1.80.2/doc'
make[2]: Entering directory `/root/ossfs-1.80.2'
make[2]: Leaving directory `/root/ossfs-1.80.2'
make[1]: Leaving directory `/root/ossfs-1.80.2'
Making install in src
make[1]: Entering directory `/root/ossfs-1.80.2/src'
make[2]: Entering directory `/root/ossfs-1.80.2/src'
 /usr/bin/mkdir -p '/usr/local/bin'
  /usr/bin/install -c ossfs '/usr/local/bin'
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/root/ossfs-1.80.2/src'
make[1]: Leaving directory `/root/ossfs-1.80.2/src'
Making install in test
make[1]: Entering directory `/root/ossfs-1.80.2/test'
make[2]: Entering directory `/root/ossfs-1.80.2/test'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/root/ossfs-1.80.2/test'
make[1]: Leaving directory `/root/ossfs-1.80.2/test'
Making install in doc
make[1]: Entering directory `/root/ossfs-1.80.2/doc'
make[2]: Entering directory `/root/ossfs-1.80.2/doc'
make[2]: Nothing to be done for `install-exec-am'.
 /usr/bin/mkdir -p '/usr/local/share/man/man1'
 /usr/bin/install -c -m 644 man/ossfs.1 '/usr/local/share/man/man1'
make[2]: Leaving directory `/root/ossfs-1.80.2/doc'
make[1]: Leaving directory `/root/ossfs-1.80.2/doc'
make[1]: Entering directory `/root/ossfs-1.80.2'
make[2]: Entering directory `/root/ossfs-1.80.2'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/root/ossfs-1.80.2'
make[1]: Leaving directory `/root/ossfs-1.80.2'

验证

[root@tokyo ~]# ossfs --version
Aliyun Open Storage Service File System V1.80.2(commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
Copyright (C) 2015 Haoran Yang <yangzhuodog1982@gmail.com>
License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[root@tokyo ~]# which ossfs
/usr/local/bin/ossfs
[root@tokyo ~]# 

使用

[root@tokyo ossfs-1.80.2]# echo bucketname :AK :SK > /etc/passwd-ossfs

[root@tokyo ossfs-1.80.2]# chmod 640 /etc/passwd-
passwd-       passwd-ossfs  

[root@tokyo ossfs-1.80.2]# chmod 640 /etc/passwd-ossfs 

[root@tokyo ossfs-1.80.2]# mkdir /tmp/ossfs

[root@tokyo ossfs-1.80.2]# ossfs bucketname /tmp/ossfs/ -ourl=http://oss-cn-shanghai.aliyuncs.com

[root@tokyo ossfs-1.80.2]# df -Th
Filesystem     Type        Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs    486M     0  486M   0% /dev
tmpfs          tmpfs       497M     0  497M   0% /dev/shm
tmpfs          tmpfs       497M   51M  447M  11% /run
tmpfs          tmpfs       497M     0  497M   0% /sys/fs/cgroup
/dev/vda1      ext4         25G   14G   11G  57% /
tmpfs          tmpfs       100M     0  100M   0% /run/user/0
ossfs          fuse.ossfs  256T     0  256T   0% /tmp/ossfs

[root@tokyo ossfs-1.80.2]# cd /tmp/ossfs/

[root@tokyo ossfs]# ls

[root@tokyo ossfs]# echo hello > a.txt

[root@tokyo ossfs]# ls
a.txt
[root@tokyo ossfs]#

局限性

ossfs提供的功能和性能和本地文件系统相比,具有一些局限性。具体包括:

  • 随机或者追加写文件会导致整个文件的重写。
  • 元数据操作,例如list directory,性能较差,因为需要远程访问oss服务器。
  • 文件/文件夹的rename操作不是原子的。
  • 多个客户端挂载同一个oss bucket时,依赖用户自行协调各个客户端的行为。例如避免多个客户端写同一个文件等等。
  • 不支持hard link。
  • 不适合用在高并发读/写的场景,这样会让系统的load升高

相关链接


Flyfish's Blog - 飞鱼博客| 点滴记忆,记录成长---flyfish