README.md (14394B)
1  2 3 A FileSystem Abstraction System for Go 4 5 [](https://github.com/spf13/afero/actions/workflows/test.yml) [](https://godoc.org/github.com/spf13/afero) [](https://gitter.im/spf13/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) 6 7 # Overview 8 9 Afero is a filesystem framework providing a simple, uniform and universal API 10 interacting with any filesystem, as an abstraction layer providing interfaces, 11 types and methods. Afero has an exceptionally clean interface and simple design 12 without needless constructors or initialization methods. 13 14 Afero is also a library providing a base set of interoperable backend 15 filesystems that make it easy to work with afero while retaining all the power 16 and benefit of the os and ioutil packages. 17 18 Afero provides significant improvements over using the os package alone, most 19 notably the ability to create mock and testing filesystems without relying on the disk. 20 21 It is suitable for use in any situation where you would consider using the OS 22 package as it provides an additional abstraction that makes it easy to use a 23 memory backed file system during testing. It also adds support for the http 24 filesystem for full interoperability. 25 26 27 ## Afero Features 28 29 * A single consistent API for accessing a variety of filesystems 30 * Interoperation between a variety of file system types 31 * A set of interfaces to encourage and enforce interoperability between backends 32 * An atomic cross platform memory backed file system 33 * Support for compositional (union) file systems by combining multiple file systems acting as one 34 * Specialized backends which modify existing filesystems (Read Only, Regexp filtered) 35 * A set of utility functions ported from io, ioutil & hugo to be afero aware 36 * Wrapper for go 1.16 filesystem abstraction `io/fs.FS` 37 38 # Using Afero 39 40 Afero is easy to use and easier to adopt. 41 42 A few different ways you could use Afero: 43 44 * Use the interfaces alone to define your own file system. 45 * Wrapper for the OS packages. 46 * Define different filesystems for different parts of your application. 47 * Use Afero for mock filesystems while testing 48 49 ## Step 1: Install Afero 50 51 First use go get to install the latest version of the library. 52 53 $ go get github.com/spf13/afero 54 55 Next include Afero in your application. 56 ```go 57 import "github.com/spf13/afero" 58 ``` 59 60 ## Step 2: Declare a backend 61 62 First define a package variable and set it to a pointer to a filesystem. 63 ```go 64 var AppFs = afero.NewMemMapFs() 65 66 or 67 68 var AppFs = afero.NewOsFs() 69 ``` 70 It is important to note that if you repeat the composite literal you 71 will be using a completely new and isolated filesystem. In the case of 72 OsFs it will still use the same underlying filesystem but will reduce 73 the ability to drop in other filesystems as desired. 74 75 ## Step 3: Use it like you would the OS package 76 77 Throughout your application use any function and method like you normally 78 would. 79 80 So if my application before had: 81 ```go 82 os.Open("/tmp/foo") 83 ``` 84 We would replace it with: 85 ```go 86 AppFs.Open("/tmp/foo") 87 ``` 88 89 `AppFs` being the variable we defined above. 90 91 92 ## List of all available functions 93 94 File System Methods Available: 95 ```go 96 Chmod(name string, mode os.FileMode) : error 97 Chown(name string, uid, gid int) : error 98 Chtimes(name string, atime time.Time, mtime time.Time) : error 99 Create(name string) : File, error 100 Mkdir(name string, perm os.FileMode) : error 101 MkdirAll(path string, perm os.FileMode) : error 102 Name() : string 103 Open(name string) : File, error 104 OpenFile(name string, flag int, perm os.FileMode) : File, error 105 Remove(name string) : error 106 RemoveAll(path string) : error 107 Rename(oldname, newname string) : error 108 Stat(name string) : os.FileInfo, error 109 ``` 110 File Interfaces and Methods Available: 111 ```go 112 io.Closer 113 io.Reader 114 io.ReaderAt 115 io.Seeker 116 io.Writer 117 io.WriterAt 118 119 Name() : string 120 Readdir(count int) : []os.FileInfo, error 121 Readdirnames(n int) : []string, error 122 Stat() : os.FileInfo, error 123 Sync() : error 124 Truncate(size int64) : error 125 WriteString(s string) : ret int, err error 126 ``` 127 In some applications it may make sense to define a new package that 128 simply exports the file system variable for easy access from anywhere. 129 130 ## Using Afero's utility functions 131 132 Afero provides a set of functions to make it easier to use the underlying file systems. 133 These functions have been primarily ported from io & ioutil with some developed for Hugo. 134 135 The afero utilities support all afero compatible backends. 136 137 The list of utilities includes: 138 139 ```go 140 DirExists(path string) (bool, error) 141 Exists(path string) (bool, error) 142 FileContainsBytes(filename string, subslice []byte) (bool, error) 143 GetTempDir(subPath string) string 144 IsDir(path string) (bool, error) 145 IsEmpty(path string) (bool, error) 146 ReadDir(dirname string) ([]os.FileInfo, error) 147 ReadFile(filename string) ([]byte, error) 148 SafeWriteReader(path string, r io.Reader) (err error) 149 TempDir(dir, prefix string) (name string, err error) 150 TempFile(dir, prefix string) (f File, err error) 151 Walk(root string, walkFn filepath.WalkFunc) error 152 WriteFile(filename string, data []byte, perm os.FileMode) error 153 WriteReader(path string, r io.Reader) (err error) 154 ``` 155 For a complete list see [Afero's GoDoc](https://godoc.org/github.com/spf13/afero) 156 157 They are available under two different approaches to use. You can either call 158 them directly where the first parameter of each function will be the file 159 system, or you can declare a new `Afero`, a custom type used to bind these 160 functions as methods to a given filesystem. 161 162 ### Calling utilities directly 163 164 ```go 165 fs := new(afero.MemMapFs) 166 f, err := afero.TempFile(fs,"", "ioutil-test") 167 168 ``` 169 170 ### Calling via Afero 171 172 ```go 173 fs := afero.NewMemMapFs() 174 afs := &afero.Afero{Fs: fs} 175 f, err := afs.TempFile("", "ioutil-test") 176 ``` 177 178 ## Using Afero for Testing 179 180 There is a large benefit to using a mock filesystem for testing. It has a 181 completely blank state every time it is initialized and can be easily 182 reproducible regardless of OS. You could create files to your heart’s content 183 and the file access would be fast while also saving you from all the annoying 184 issues with deleting temporary files, Windows file locking, etc. The MemMapFs 185 backend is perfect for testing. 186 187 * Much faster than performing I/O operations on disk 188 * Avoid security issues and permissions 189 * Far more control. 'rm -rf /' with confidence 190 * Test setup is far more easier to do 191 * No test cleanup needed 192 193 One way to accomplish this is to define a variable as mentioned above. 194 In your application this will be set to afero.NewOsFs() during testing you 195 can set it to afero.NewMemMapFs(). 196 197 It wouldn't be uncommon to have each test initialize a blank slate memory 198 backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere 199 appropriate in my application code. This approach ensures that Tests are order 200 independent, with no test relying on the state left by an earlier test. 201 202 Then in my tests I would initialize a new MemMapFs for each test: 203 ```go 204 func TestExist(t *testing.T) { 205 appFS := afero.NewMemMapFs() 206 // create test files and directories 207 appFS.MkdirAll("src/a", 0755) 208 afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644) 209 afero.WriteFile(appFS, "src/c", []byte("file c"), 0644) 210 name := "src/c" 211 _, err := appFS.Stat(name) 212 if os.IsNotExist(err) { 213 t.Errorf("file \"%s\" does not exist.\n", name) 214 } 215 } 216 ``` 217 218 # Available Backends 219 220 ## Operating System Native 221 222 ### OsFs 223 224 The first is simply a wrapper around the native OS calls. This makes it 225 very easy to use as all of the calls are the same as the existing OS 226 calls. It also makes it trivial to have your code use the OS during 227 operation and a mock filesystem during testing or as needed. 228 229 ```go 230 appfs := afero.NewOsFs() 231 appfs.MkdirAll("src/a", 0755) 232 ``` 233 234 ## Memory Backed Storage 235 236 ### MemMapFs 237 238 Afero also provides a fully atomic memory backed filesystem perfect for use in 239 mocking and to speed up unnecessary disk io when persistence isn’t 240 necessary. It is fully concurrent and will work within go routines 241 safely. 242 243 ```go 244 mm := afero.NewMemMapFs() 245 mm.MkdirAll("src/a", 0755) 246 ``` 247 248 #### InMemoryFile 249 250 As part of MemMapFs, Afero also provides an atomic, fully concurrent memory 251 backed file implementation. This can be used in other memory backed file 252 systems with ease. Plans are to add a radix tree memory stored file 253 system using InMemoryFile. 254 255 ## Network Interfaces 256 257 ### SftpFs 258 259 Afero has experimental support for secure file transfer protocol (sftp). Which can 260 be used to perform file operations over a encrypted channel. 261 262 ### GCSFs 263 264 Afero has experimental support for Google Cloud Storage (GCS). You can either set the 265 `GOOGLE_APPLICATION_CREDENTIALS_JSON` env variable to your JSON credentials or use `opts` in 266 `NewGcsFS` to configure access to your GCS bucket. 267 268 Some known limitations of the existing implementation: 269 * No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version. 270 * No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version. 271 * Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent. 272 273 274 ## Filtering Backends 275 276 ### BasePathFs 277 278 The BasePathFs restricts all operations to a given path within an Fs. 279 The given file name to the operations on this Fs will be prepended with 280 the base path before calling the source Fs. 281 282 ```go 283 bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path") 284 ``` 285 286 ### ReadOnlyFs 287 288 A thin wrapper around the source Fs providing a read only view. 289 290 ```go 291 fs := afero.NewReadOnlyFs(afero.NewOsFs()) 292 _, err := fs.Create("/file.txt") 293 // err = syscall.EPERM 294 ``` 295 296 # RegexpFs 297 298 A filtered view on file names, any file NOT matching 299 the passed regexp will be treated as non-existing. 300 Files not matching the regexp provided will not be created. 301 Directories are not filtered. 302 303 ```go 304 fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`)) 305 _, err := fs.Create("/file.html") 306 // err = syscall.ENOENT 307 ``` 308 309 ### HttpFs 310 311 Afero provides an http compatible backend which can wrap any of the existing 312 backends. 313 314 The Http package requires a slightly specific version of Open which 315 returns an http.File type. 316 317 Afero provides an httpFs file system which satisfies this requirement. 318 Any Afero FileSystem can be used as an httpFs. 319 320 ```go 321 httpFs := afero.NewHttpFs(<ExistingFS>) 322 fileserver := http.FileServer(httpFs.Dir(<PATH>)) 323 http.Handle("/", fileserver) 324 ``` 325 326 ## Composite Backends 327 328 Afero provides the ability have two filesystems (or more) act as a single 329 file system. 330 331 ### CacheOnReadFs 332 333 The CacheOnReadFs will lazily make copies of any accessed files from the base 334 layer into the overlay. Subsequent reads will be pulled from the overlay 335 directly permitting the request is within the cache duration of when it was 336 created in the overlay. 337 338 If the base filesystem is writeable, any changes to files will be 339 done first to the base, then to the overlay layer. Write calls to open file 340 handles like `Write()` or `Truncate()` to the overlay first. 341 342 To writing files to the overlay only, you can use the overlay Fs directly (not 343 via the union Fs). 344 345 Cache files in the layer for the given time.Duration, a cache duration of 0 346 means "forever" meaning the file will not be re-requested from the base ever. 347 348 A read-only base will make the overlay also read-only but still copy files 349 from the base to the overlay when they're not present (or outdated) in the 350 caching layer. 351 352 ```go 353 base := afero.NewOsFs() 354 layer := afero.NewMemMapFs() 355 ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second) 356 ``` 357 358 ### CopyOnWriteFs() 359 360 The CopyOnWriteFs is a read only base file system with a potentially 361 writeable layer on top. 362 363 Read operations will first look in the overlay and if not found there, will 364 serve the file from the base. 365 366 Changes to the file system will only be made in the overlay. 367 368 Any attempt to modify a file found only in the base will copy the file to the 369 overlay layer before modification (including opening a file with a writable 370 handle). 371 372 Removing and Renaming files present only in the base layer is not currently 373 permitted. If a file is present in the base layer and the overlay, only the 374 overlay will be removed/renamed. 375 376 ```go 377 base := afero.NewOsFs() 378 roBase := afero.NewReadOnlyFs(base) 379 ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs()) 380 381 fh, _ = ufs.Create("/home/test/file2.txt") 382 fh.WriteString("This is a test") 383 fh.Close() 384 ``` 385 386 In this example all write operations will only occur in memory (MemMapFs) 387 leaving the base filesystem (OsFs) untouched. 388 389 390 ## Desired/possible backends 391 392 The following is a short list of possible backends we hope someone will 393 implement: 394 395 * SSH 396 * S3 397 398 # About the project 399 400 ## What's in the name 401 402 Afero comes from the latin roots Ad-Facere. 403 404 **"Ad"** is a prefix meaning "to". 405 406 **"Facere"** is a form of the root "faciō" making "make or do". 407 408 The literal meaning of afero is "to make" or "to do" which seems very fitting 409 for a library that allows one to make files and directories and do things with them. 410 411 The English word that shares the same roots as Afero is "affair". Affair shares 412 the same concept but as a noun it means "something that is made or done" or "an 413 object of a particular type". 414 415 It's also nice that unlike some of my other libraries (hugo, cobra, viper) it 416 Googles very well. 417 418 ## Release Notes 419 420 See the [Releases Page](https://github.com/spf13/afero/releases). 421 422 ## Contributing 423 424 1. Fork it 425 2. Create your feature branch (`git checkout -b my-new-feature`) 426 3. Commit your changes (`git commit -am 'Add some feature'`) 427 4. Push to the branch (`git push origin my-new-feature`) 428 5. Create new Pull Request 429 430 ## Contributors 431 432 Names in no particular order: 433 434 * [spf13](https://github.com/spf13) 435 * [jaqx0r](https://github.com/jaqx0r) 436 * [mbertschler](https://github.com/mbertschler) 437 * [xor-gate](https://github.com/xor-gate) 438 439 ## License 440 441 Afero is released under the Apache 2.0 license. See 442 [LICENSE.txt](https://github.com/spf13/afero/blob/master/LICENSE.txt)