storage.md (4975B)
1 # Storage 2 3 ## Settings 4 5 ```yaml 6 ########################## 7 ##### STORAGE CONFIG ##### 8 ########################## 9 10 # Config pertaining to storage of user-created uploads (videos, images, etc). 11 12 # String. Type of storage backend to use. 13 # Examples: ["local", "s3"] 14 # Default: "local" (storage on local disk) 15 storage-backend: "local" 16 17 # String. Directory to use as a base path for storing files. 18 # Make sure whatever user/group gotosocial is running as has permission to access 19 # this directory, and create new subdirectories and files within it. 20 # Only required when running with the local storage backend. 21 # Examples: ["/home/gotosocial/storage", "/opt/gotosocial/datastorage"] 22 # Default: "/gotosocial/storage" 23 storage-local-base-path: "/gotosocial/storage" 24 25 # String. API endpoint of the S3 compatible service. 26 # Only required when running with the s3 storage backend. 27 # 28 # If your endpoint contains the bucket name, all files will be put into a 29 # subdirectory with the name of `storage-s3-bucket` 30 # 31 # Examples: ["minio:9000", "s3.nl-ams.scw.cloud", "s3.us-west-002.backblazeb2.com"] 32 # Default: "" 33 storage-s3-endpoint: "" 34 35 # Bool. If data stored in S3 should be proxied through GoToSocial instead of redirecting to a presigned URL. 36 # 37 # Default: false 38 storage-s3-proxy: false 39 # Bool. Use SSL for S3 connections. 40 # 41 # Only set this to 'false' when testing locally. 42 # 43 # Default: true 44 storage-s3-use-ssl: true 45 46 # String. Access key part of the S3 credentials. 47 # Consider setting this value using environment variables to avoid leaking it via the config file 48 # Only required when running with the s3 storage backend. 49 # Examples: ["AKIAJSIE27KKMHXI3BJQ","miniouser"] 50 # Default: "" 51 storage-s3-access-key: "" 52 # String. Secret key part of the S3 credentials. 53 # Consider setting this value using environment variables to avoid leaking it via the config file 54 # Only required when running with the s3 storage backend. 55 # Examples: ["5bEYu26084qjSFyclM/f2pz4gviSfoOg+mFwBH39","miniopassword"] 56 # Default: "" 57 storage-s3-secret-key: "" 58 # String. Name of the storage bucket. 59 # 60 # If you have already encoded your bucket name in the storage-s3-endpoint, this 61 # value will be used as a directory containing your data. 62 # 63 # The bucket must exist prior to starting GoToSocial 64 # 65 # Only required when running with the s3 storage backend. 66 # Examples: ["gts","cool-instance"] 67 # Default: "" 68 storage-s3-bucket: "" 69 ``` 70 71 ### AWS S3 Bucket Configuration 72 73 #### Bucket Created 74 GoToSocial by default creates signed URL's which means we dont need to change anything major on the policies of the bucket. 75 Here are the steps to follow for bucket creation 76 77 1. Login to AWS -> select S3 as service. 78 2. click Create Bucket 79 3. Provide a unique name and avoid adding "." in the name 80 4. Do not change the public access settings (Let them be on "block public access" mode) 81 82 #### AWS ACCESS KEY Configuration 83 84 1. In AWS Console -> IAM (under Security, Identity, & Compliance) 85 2. Add a user with programatic api's access 86 3. We recommend setting up below listed policy, replace <bucketname> with your buckets name 87 88 ```json 89 { 90 "Statement": [ 91 { 92 "Effect": "Allow", 93 "Action": "s3:ListAllMyBuckets", 94 "Resource": "arn:aws:s3:::*" 95 }, 96 { 97 "Effect": "Allow", 98 "Action": "s3:*", 99 "Resource": [ 100 "arn:aws:s3:::<bucket_name>", 101 "arn:aws:s3:::<bucket_name>/*" 102 ] 103 } 104 ] 105 } 106 ``` 107 108 4. Provide the values in config above 109 110 * storage-s3-endpoint -> should be your bucket location say `s3.ap-southeast-1.amazonaws.com` 111 * storage-s3-access-key -> Access key you obtained for the user created above 112 * storage-s3-secret-key -> Secret key you obtained for the user created above 113 * storage-s3-bucket -> Keep this as the <bucketname> that you created just now. 114 115 116 117 #### Migrating data from local storage to AWS s3 bucket 118 119 This step is only needed if you have a running instance. Ignore this if you are setting up a fresh instance. 120 We have provided [s3cmd](https://github.com/s3tools/s3cmd) command for the copy operation. 121 122 ```bash 123 s3cmd sync --add-header="Cache-Control:public, max-age=315576000, immutable" ./ s3://<bucket name> 124 ``` 125 126 127 ### Migrating between backends 128 129 Currently, migration between backends is freely possible. To do so, you only 130 have to move the directories (and their contents) between the different implementations. 131 132 One way to do so, is by utilizing the [MinIO 133 Client](https://docs.min.io/docs/minio-client-complete-guide.html). The 134 migration process might look something like this: 135 136 ```bash 137 # 1. Change the GoToSocial configuration to the new backend (and restart) 138 # 2. Register the S3 Backend with the MinIO client 139 mc alias set scw https://s3.nl-ams.scw.cloud 140 # 3. Mirror the folder structure to the remote bucket 141 mc mirror /gotosocial/storage/ scw/example-bucket/ 142 # 4. Aaaand we're done! 143 ``` 144 145 If you want to migrate back, switch around the arguments of the `mc mirror` command.