3大クラウドでVMとストレージを組み合わせた構成を一撃で構築する
EC2+S3でSambaサーバ組み合わせてファイルサーバ的なものを前回やってみましたが、GCPとかAzureでも同じことやってみました。
GCPの場合はGCE+GCSにgcsfuseを組み合わせて構成、Azureの場合はAzureVMにBlobの機能でNFSv3を指定してやるやり方です。
Azureは以前にredmineでAzureFiles使ってマウントしたことがありましたが今回はBlobのコンテナ使ってNFSマウントしてみました。
AWSの場合はgoofysがパフォーマンスいいみたいです。GCPとAzureも書込みとか変更は我慢できないほどではないけどファイル削除に若干、時間がかかる。
頻繁な削除が発生しないなら、まぁどれでもいいんじゃなかろうかという感じです。コストはあまり見ていないですけどVMのコスト+ストレージのコスト+読み書き発生するみたいですね。
◆Shellスクリプト実行環境の前提
・AWS：IAMロールでS3フルアクセス可能なものが存在しEC2に付与されている。AWS-CLI用のクレデンシャルが発行済
・GCP：GCEのアクセススコープで[ストレージ]がフル設定となっている
・Azure：V-NETとストレージアカウントが設定されていること
・Ubuntu22.04LTSにて確認(ロケールと言語は日本語に変換済)
◆Shell本体
#!/bin/bash
ACCESS_KEY_ID=AKXXXXX
ACCOUNT_ACCESS_KEY_ID=XXXXX
ACCOUNT_REGION_ID=ap-northeast-1
ACCOUNT_OUTPUT=json
S3_BUCKET_MP_PUBLIC=s3test-mountpoint-samba
S3_BUCKET_S3FS=s3test-s3fs-samba
S3_BUCKET_GFYS=s3test-goofys-samba
GCS_BUCKET=gcstest-samba
AZURE_STORAGE_ACOUNT=azuresambatest
AZURE_CONTAINER=azure-samba-test
MOUNT_DIR_MPS3=/mnt/test-mountpoint-s3
MOUNT_DIR_S3FS=/mnt/test-s3fs
MOUNT_DIR_GFYS=/mnt/test-goofys
MOUNT_DIR_GCS=/mnt/test-gcs
MOUNT_DIR_AZURE=/mnt/test-azure
IAM_ROLE=your-iam-role
URL=https://s3-ap-northeast-1.amazonaws.com
WORKGROUP=WORKGROUP
PUBLIC_DIR=public
PUBLIC_SHARE_MPS3=public_share_mountpoint
PUBLIC_SHARE_S3FS=public_share_s3fs
PUBLIC_SHARE_GFYS=public_share_goofys
PUBLIC_SHARE_GCS=public_share_gcs
PUBLIC_SHARE_AZURE=public_share_azure
CONTENTS_DIR=contents
CONTENTS_SHARE_MPS3=contents_share_mountpoint
CONTENTS_SHARE_S3FS=contents_share_s3fs
CONTENTS_SHARE_GFYS=contents_share_goofys
CONTENTS_SHARE_GCS=contents_share_gcs
CONTENTS_SHARE_AZURE=contents_share_azure
ADMIN_GRP=admin_group
CONTENTS_MGR_GRP=content_manager_group
VIEWER_GRP=viewer_group
ADMIN_USER=smb_sysadm
CONTENTS_MGR=smb_contentsadm
VIEW_USER=smb_viewuser1
ADMIN_PASSWORD=password
CNT_PASSWORD=password
VIEW_PASSWORD=password
#
# 1-1.AWS用マウント設定
#
read -p "## Do you want to install AWS mount tools? (y/n) [y]: " -e -i "y" awsS3mountResponse
if [ "$awsS3mountResponse" == "y" ] || [ "$awsS3mountResponse" == "Y" ]; then
# バケット作成
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo apt -y install unzip
unzip awscliv2.zip
sudo ./aws/install
mkdir -p .aws
rm awscliv2.zip
tee .aws/credentials <<_EOF_
[default]
aws_access_key_id = ${ACCESS_KEY_ID}
aws_secret_access_key = ${ACCOUNT_ACCESS_KEY_ID}
_EOF_
tee .aws/config <<_EOF_
[default]
region = ${ACCOUNT_REGION_ID}
output = ${ACCOUNT_OUTPUT}
_EOF_
aws s3 mb s3://${S3_BUCKET_MP_PUBLIC}
aws s3 mb s3://${S3_BUCKET_S3FS}
aws s3 mb s3://${S3_BUCKET_GFYS}
# 1.Mountpoint for Amazon S3のインストール
sudo mkdir ${MOUNT_DIR_MPS3}
sudo wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
sudo apt-get install -y ./mount-s3.deb
sudo tee /etc/systemd/system/mount-s3.service <<_EOF_
[Unit]
Description=Mountpoint for Amazon S3 mount
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=mount-s3 --allow-delete --allow-other --dir-mode 0777 ${S3_BUCKET_MP_PUBLIC} ${MOUNT_DIR_MPS3}
ExecStop=/usr/bin/fusermount -u /mnt
[Install]
WantedBy=default.target
_EOF_
sudo chmod 755 /lib/systemd/system/mount-s3.service
sudo systemctl start mount-s3.service
sudo systemctl enable mount-s3.service
rm -Rf mount-s3.deb
# 2.S3fsインストール
sudo mkdir ${MOUNT_DIR_S3FS}
sudo apt -y update
sudo apt -y install s3fs
echo "/usr/bin/s3fs#${S3_BUCKET_S3FS} ${MOUNT_DIR_S3FS} fuse _netdev,iam_role=${IAM_ROLE},url=${URL},allow_other 0 0" | sudo tee -a /etc/fstab
rm -Rf s3fs-fuse/
#sudo /usr/bin/s3fs ${S3_BUCKET_S3FS} ${MOUNT_DIR_S3FS} -o iam_role=${IAM_ROLE},url=${URL},allow_other
sudo mount -a
# 3.goofysインストール
sudo mkdir ${MOUNT_DIR_GFYS}
sudo apt -y update
sudo apt -y install fuse
GOOFYS_VERSION=$(curl -s https://api.github.com/repos/kahing/goofys/releases/latest | grep tag_name | cut -d '"' -f 4)
sudo wget https://github.com/kahing/goofys/releases/download/v0.24.0/goofys -P /usr/local/bin/
sudo chmod 755 /usr/local/bin/goofys
echo "/usr/local/bin/goofys#${S3_BUCKET_GFYS} ${MOUNT_DIR_GFYS} fuse _netdev,allow_other,--dir-mode=0777,--file-mode=0666 0 0" | sudo tee -a /etc/fstab
#sudo goofys s3test-goofys-samba /mnt/test-goofys/
sudo mount -a
fi
#
# 1-2.GCP用マウント設定
#
read -p "## Do you want to install Google mount tools? (y/n) [y]: " -e -i "n" gcsmountResponse
if [ "$gcsmountResponse" == "y" ] || [ "$gcsmountResponse" == "Y" ]; then
# バケット作成
gcloud storage buckets create gs://${GCS_BUCKET} --default-storage-class=standard --location=asia-northeast1 --uniform-bucket-level-access
# gcsfuseインストール
export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo deb https://packages.cloud.google.com/apt $GCSFUSE_REPO main | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt -y update
sudo apt -y install fuse gcsfuse
gcloud auth application-default login
sudo mkdir ${MOUNT_DIR_GCS}
echo "${GCS_BUCKET} ${MOUNT_DIR_GCS} gcsfuse rw,_netdev,allow_other,file_mode=777,dir_mode=777" | sudo tee -a /etc/fstab
sudo mount -a
fi
#
# 1-3.Azure用マウント設定
#
read -p "## Do you want to install Azure mount tools? (y/n) [y]: " -e -i "n" azuremountResponse
if [ "$azuremountResponse" == "y" ] || [ "$azuremountResponse" == "Y" ]; then
# ストレージアカウントとblobコンテナの作成
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
az login
az storage container create \
--name ${AZURE_CONTAINER} \
--account-name ${AZURE_STORAGE_ACOUNT}
# NFSv3でコンテナをマウント
sudo mkdir ${MOUNT_DIR_AZURE}
sudo apt -y update
sudo apt -y install nfs-common
echo "${AZURE_STORAGE_ACOUNT}.blob.core.windows.net:/${AZURE_STORAGE_ACOUNT}/${AZURE_CONTAINER} ${MOUNT_DIR_AZURE} nfs sec=sys,vers=3,nolock,proto=tcp 0 0" | sudo tee -a /etc/fstab
sudo mount -a
fi
#
# 2.Sambaインストール
#
read -p "## Do you want to install samba? (y/n) [y]: " -e -i "y" sambaResponse
if [ "$sambaResponse" == "y" ] || [ "$sambaResponse" == "Y" ]; then
sudo apt -y update
sudo apt -y install samba
sudo cp -r /etc/samba/smb.conf /etc/samba/smb.conf.org
sudo tee /etc/samba/smb.conf <<_EOF_
[global]
unix charset = UTF-8
dos charset = CP932
workgroup = WORKGROUP
; interfaces = 127.0.0.0/8 eth0
; bind interfaces only = yes
max log size = 1000
logging = file
log file = /var/log/samba/log.audit
vfs objects = full_audit
full_audit:prefix = %u|%I|%m|%S
full_audit:success = all
full_audit:failure = all
panic action = /usr/share/samba/panic-action %d
server role = standalone server
obey pam restrictions = yes
unix password sync = yes
passwd program = /usr/bin/passwd %u
passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
pam password change = yes
; map to guest = bad user
security = user
usershare allow guests = yes
_EOF_
if [ "$awsS3mountResponse" == "y" ] || [ "$awsS3mountResponse" == "Y" ]; then
# Samba設定ファイル準備
sudo mkdir -p ${MOUNT_DIR_MPS3}/${PUBLIC_DIR}/
sudo mkdir -p ${MOUNT_DIR_MPS3}/${CONTENTS_DIR}/
sudo chmod -R 777 ${MOUNT_DIR_MPS3}/
sudo mkdir -p ${MOUNT_DIR_S3FS}/${PUBLIC_DIR}/
sudo mkdir -p ${MOUNT_DIR_S3FS}/${CONTENTS_DIR}/
sudo chmod -R 777 ${MOUNT_DIR_S3FS}/
sudo mkdir -p ${MOUNT_DIR_GFYS}/${PUBLIC_DIR}/
sudo mkdir -p ${MOUNT_DIR_GFYS}/${CONTENTS_DIR}/
sudo chmod -R 777 ${MOUNT_DIR_GFYS}/
sudo chmod -R 777 /mnt/test-mountpoint-s3
sudo tee -a /etc/samba/smb.conf <<_EOF_
[${PUBLIC_SHARE_MPS3}]
path = ${MOUNT_DIR_MPS3}/${PUBLIC_DIR}
public = yes
writable = yes
force create mode = 0666
force directory mode = 0777
browseable = yes
[${PUBLIC_SHARE_S3FS}]
path = ${MOUNT_DIR_S3FS}/${PUBLIC_DIR}
public = yes
writable = yes
force create mode = 0666
force directory mode = 0777
browseable = yes
[${PUBLIC_SHARE_GFYS}]
path = ${MOUNT_DIR_GFYS}/${PUBLIC_DIR}
public = yes
writable = yes
force create mode = 0666
force directory mode = 0777
browseable = yes
[${CONTENTS_SHARE_MPS3}]
comment = Viewer Share
path = ${MOUNT_DIR_MPS3}/${CONTENTS_DIR}
browseable = yes
read only = yes
public = yes
guest ok = no
valid users = @${ADMIN_GRP} @${CONTENTS_MGR_GRP} @${VIEWER_GRP}
write list = @${ADMIN_GRP} @${CONTENTS_MGR_GRP}
force group = ${CONTENTS_MGR_GRP}
create mask = 0660
directory mask = 0770
[${CONTENTS_SHARE_S3FS}]
comment = Viewer Share
path = ${MOUNT_DIR_S3FS}/${CONTENTS_DIR}
browseable = yes
read only = yes
public = yes
guest ok = no
valid users = @${ADMIN_GRP} @${CONTENTS_MGR_GRP} @${VIEWER_GRP}
write list = @${ADMIN_GRP} @${CONTENTS_MGR_GRP}
force group = ${CONTENTS_MGR_GRP}
create mask = 0660
directory mask = 0770
[${CONTENTS_SHARE_GFYS}]
comment = Viewer Share
path = ${MOUNT_DIR_GFYS}/${CONTENTS_DIR}
browseable = yes
read only = yes
public = yes
guest ok = no
valid users = @${ADMIN_GRP} @${CONTENTS_MGR_GRP} @${VIEWER_GRP}
write list = @${ADMIN_GRP} @${CONTENTS_MGR_GRP}
force group = ${CONTENTS_MGR_GRP}
create mask = 0660
directory mask = 0770
_EOF_
fi
# GCS用
if [ "$gcsmountResponse" == "y" ] || [ "$gcsmountResponse" == "Y" ]; then
sudo mkdir -p ${MOUNT_DIR_GCS}/${PUBLIC_DIR}/
sudo mkdir -p ${MOUNT_DIR_GCS}/${CONTENTS_DIR}/
sudo chmod -R 777 ${MOUNT_DIR_GCS}/
sudo tee -a /etc/samba/smb.conf <<_EOF_
[${PUBLIC_SHARE_GCS}]
path = ${MOUNT_DIR_GCS}/${PUBLIC_DIR}
public = yes
writable = yes
force create mode = 0666
force directory mode = 0777
browseable = yes
[${CONTENTS_SHARE_GCS}]
comment = Viewer Share
path = ${MOUNT_DIR_GCS}/${CONTENTS_DIR}
browseable = yes
read only = yes
public = yes
guest ok = no
valid users = @${ADMIN_GRP} @${CONTENTS_MGR_GRP} @${VIEWER_GRP}
write list = @${ADMIN_GRP} @${CONTENTS_MGR_GRP}
force group = ${CONTENTS_MGR_GRP}
create mask = 0660
directory mask = 0770
_EOF_
fi
# Azure用
if [ "$azuremountResponse" == "y" ] || [ "$azuremountResponse" == "Y" ]; then
sudo mkdir -p ${MOUNT_DIR_AZURE}/${PUBLIC_DIR}/
sudo mkdir -p ${MOUNT_DIR_AZURE}/${CONTENTS_DIR}/
sudo chmod -R 777 ${MOUNT_DIR_AZURE}/
sudo tee -a /etc/samba/smb.conf <<_EOF_
[${PUBLIC_SHARE_AZURE}]
path = ${MOUNT_DIR_AZURE}/${PUBLIC_DIR}
public = yes
writable = yes
force create mode = 0666
force directory mode = 0777
browseable = yes
[${CONTENTS_SHARE_AZURE}]
comment = Viewer Share
path = ${MOUNT_DIR_AZURE}/${CONTENTS_DIR}
browseable = yes
read only = yes
public = yes
guest ok = no
valid users = @${ADMIN_GRP} @${CONTENTS_MGR_GRP} @${VIEWER_GRP}
write list = @${ADMIN_GRP} @${CONTENTS_MGR_GRP}
force group = ${CONTENTS_MGR_GRP}
create mask = 0660
directory mask = 0770
_EOF_
fi
# samba再起動
sudo systemctl restart smbd
fi
#
# 2.Webmin
#
read -p "## Do you want to install webmin? (y/n) [y]: " -e -i "y" webminResponse
if [ "$webminResponse" == "y" ] || [ "$webminResponse" == "Y" ]; then
sudo apt -y install python3 shared-mime-info unzip apt-show-versions libapt-pkg-perl libauthen-pam-perl libio-pty-perl libnet-ssleay-perl
curl -L -O https://www.webmin.com/download/deb/webmin-current.deb
sudo dpkg -i webmin-current.deb
rm webmin-current.deb
sudo systemctl enable webmin
fi
#
# 3.Group追加・ユーザ追加
#
read -p "## Do you want to linux user and sambauser? (y/n) [y]: " -e -i "y" useraddResponse
if [ "$useraddResponse" == "y" ] || [ "$useraddResponse" == "Y" ]; then
sudo addgroup ${ADMIN_GRP}
sudo addgroup ${CONTENTS_MGR_GRP}
sudo addgroup ${VIEWER_GRP}
sudo adduser --disabled-login --ingroup ${ADMIN_GRP} --disabled-password --no-create-home --gecos "" ${ADMIN_USER}
sudo adduser --disabled-login --ingroup ${CONTENTS_MGR_GRP} --disabled-password --no-create-home --gecos "" ${CONTENTS_MGR}
sudo adduser --disabled-login --ingroup ${VIEWER_GRP} --disabled-password --no-create-home --gecos "" ${VIEW_USER}
echo -e "${ADMIN_PASSWORD}\n${ADMIN_PASSWORD}" | sudo smbpasswd -a ${ADMIN_USER}
echo -e "${CNT_PASSWORD}\n${CNT_PASSWORD}" | sudo smbpasswd -a ${CONTENTS_MGR}
echo -e "${VIEW_PASSWORD}\n${VIEW_PASSWORD}" | sudo smbpasswd -a ${VIEW_USER}
fi
◆参考
・GCP
https://qiita.com/ha_shio/items/04ad7f196f6f6b1323aa
https://dev.classmethod.jp/articles/google-cloud-gce-to-gcs/
https://zenn.dev/google_cloud_jp/articles/cloudrun-gcs-fuse
https://zenn.dev/cloud_ace/articles/d8ad28fecfb531
https://kihor.com/gce-gcs-mount/
https://teech-lab.com/gcs-googlecloudstorage-mount/1345/
https://cloud.google.com/storage/docs/gcsfuse-mount?hl=ja
https://qiita.com/yoshifuji/items/ebfc21edf9bd3db88a6f
・AzureBlob
https://www.michikusayan.com/entry/20211203/1638500400
https://qiita.com/dg4101/items/9c923954e602f4697706
https://www.tama-negi.com/2021/06/20/azure-files-nfs-centos-mount/
https://qiita.com/tokawa-ms/items/f2c625228d34e413b71f
https://business.ntt-east.co.jp/content/cloudsolution/column-448.html
https://qiita.com/mikanchaaan/items/dfa353c11d27b07b99b4
https://tech-lab.sios.jp/archives/205
https://tech-blog.cloud-config.jp/2021-07-20-linux-on-azure-azure-files-permission/
https://azure-recipe.kc-cloud.jp/azure-files-for-ubuntu/
https://qiita.com/sengoku/items/e078992f0369edbb827c
https://onlyutkarsh.medium.com/how-to-create-nfs-share-azure-blob-storage-dc1a23f6768f
くだらないことひらめいたのですが、Redmineの添付ファイル保管するところディレクトリマウントして、そこを見に行くように指定してやればどこのクラウドでもストレージの心配なく使えるんじゃないかなと思いました。
そしてマウントシリーズはいったんこれで終わりです。
コメント