doc.go (8867B)
1 // Copyright (c) 2012-2020 Ugorji Nwoke. All rights reserved. 2 // Use of this source code is governed by a MIT license found in the LICENSE file. 3 4 /* 5 Package codec provides a 6 High Performance, Feature-Rich Idiomatic Go 1.4+ codec/encoding library 7 for binc, msgpack, cbor, json. 8 9 Supported Serialization formats are: 10 11 - msgpack: https://github.com/msgpack/msgpack 12 - binc: http://github.com/ugorji/binc 13 - cbor: http://cbor.io http://tools.ietf.org/html/rfc7049 14 - json: http://json.org http://tools.ietf.org/html/rfc7159 15 - simple: 16 17 This package will carefully use 'package unsafe' for performance reasons in specific places. 18 You can build without unsafe use by passing the safe or appengine tag 19 i.e. 'go install -tags=codec.safe ...'. 20 21 This library works with both the standard `gc` and the `gccgo` compilers. 22 23 For detailed usage information, read the primer at http://ugorji.net/blog/go-codec-primer . 24 25 The idiomatic Go support is as seen in other encoding packages in 26 the standard library (ie json, xml, gob, etc). 27 28 Rich Feature Set includes: 29 30 - Simple but extremely powerful and feature-rich API 31 - Support for go 1.4 and above, while selectively using newer APIs for later releases 32 - Excellent code coverage ( > 90% ) 33 - Very High Performance. 34 Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X. 35 - Careful selected use of 'unsafe' for targeted performance gains. 36 - 100% safe mode supported, where 'unsafe' is not used at all. 37 - Lock-free (sans mutex) concurrency for scaling to 100's of cores 38 - In-place updates during decode, with option to zero value in maps and slices prior to decode 39 - Coerce types where appropriate 40 e.g. decode an int in the stream into a float, decode numbers from formatted strings, etc 41 - Corner Cases: 42 Overflows, nil maps/slices, nil values in streams are handled correctly 43 - Standard field renaming via tags 44 - Support for omitting empty fields during an encoding 45 - Encoding from any value and decoding into pointer to any value 46 (struct, slice, map, primitives, pointers, interface{}, etc) 47 - Extensions to support efficient encoding/decoding of any named types 48 - Support encoding.(Binary|Text)(M|Unm)arshaler interfaces 49 - Support using existence of `IsZero() bool` to determine if a value is a zero value. 50 Analogous to time.Time.IsZero() bool. 51 - Decoding without a schema (into a interface{}). 52 Includes Options to configure what specific map or slice type to use 53 when decoding an encoded list or map into a nil interface{} 54 - Mapping a non-interface type to an interface, so we can decode appropriately 55 into any interface type with a correctly configured non-interface value. 56 - Encode a struct as an array, and decode struct from an array in the data stream 57 - Option to encode struct keys as numbers (instead of strings) 58 (to support structured streams with fields encoded as numeric codes) 59 - Comprehensive support for anonymous fields 60 - Fast (no-reflection) encoding/decoding of common maps and slices 61 - Code-generation for faster performance, supported in go 1.6+ 62 - Support binary (e.g. messagepack, cbor) and text (e.g. json) formats 63 - Support indefinite-length formats to enable true streaming 64 (for formats which support it e.g. json, cbor) 65 - Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes. 66 This mostly applies to maps, where iteration order is non-deterministic. 67 - NIL in data stream decoded as zero value 68 - Never silently skip data when decoding. 69 User decides whether to return an error or silently skip data when keys or indexes 70 in the data stream do not map to fields in the struct. 71 - Detect and error when encoding a cyclic reference (instead of stack overflow shutdown) 72 - Encode/Decode from/to chan types (for iterative streaming support) 73 - Drop-in replacement for encoding/json. `json:` key in struct tag supported. 74 - Provides a RPC Server and Client Codec for net/rpc communication protocol. 75 - Handle unique idiosyncrasies of codecs e.g. 76 For messagepack, configure how ambiguities in handling raw bytes are resolved and 77 provide rpc server/client codec to support 78 msgpack-rpc protocol defined at: 79 https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md 80 81 # Extension Support 82 83 Users can register a function to handle the encoding or decoding of 84 their custom types. 85 86 There are no restrictions on what the custom type can be. Some examples: 87 88 type BisSet []int 89 type BitSet64 uint64 90 type UUID string 91 type MyStructWithUnexportedFields struct { a int; b bool; c []int; } 92 type GifImage struct { ... } 93 94 As an illustration, MyStructWithUnexportedFields would normally be 95 encoded as an empty map because it has no exported fields, while UUID 96 would be encoded as a string. However, with extension support, you can 97 encode any of these however you like. 98 99 There is also seamless support provided for registering an extension (with a tag) 100 but letting the encoding mechanism default to the standard way. 101 102 # Custom Encoding and Decoding 103 104 This package maintains symmetry in the encoding and decoding halfs. 105 We determine how to encode or decode by walking this decision tree 106 107 - is there an extension registered for the type? 108 - is type a codec.Selfer? 109 - is format binary, and is type a encoding.BinaryMarshaler and BinaryUnmarshaler? 110 - is format specifically json, and is type a encoding/json.Marshaler and Unmarshaler? 111 - is format text-based, and type an encoding.TextMarshaler and TextUnmarshaler? 112 - else we use a pair of functions based on the "kind" of the type e.g. map, slice, int64, etc 113 114 This symmetry is important to reduce chances of issues happening because the 115 encoding and decoding sides are out of sync e.g. decoded via very specific 116 encoding.TextUnmarshaler but encoded via kind-specific generalized mode. 117 118 Consequently, if a type only defines one-half of the symmetry 119 (e.g. it implements UnmarshalJSON() but not MarshalJSON() ), 120 then that type doesn't satisfy the check and we will continue walking down the 121 decision tree. 122 123 # RPC 124 125 RPC Client and Server Codecs are implemented, so the codecs can be used 126 with the standard net/rpc package. 127 128 # Usage 129 130 The Handle is SAFE for concurrent READ, but NOT SAFE for concurrent modification. 131 132 The Encoder and Decoder are NOT safe for concurrent use. 133 134 Consequently, the usage model is basically: 135 136 - Create and initialize the Handle before any use. 137 Once created, DO NOT modify it. 138 - Multiple Encoders or Decoders can now use the Handle concurrently. 139 They only read information off the Handle (never write). 140 - However, each Encoder or Decoder MUST not be used concurrently 141 - To re-use an Encoder/Decoder, call Reset(...) on it first. 142 This allows you use state maintained on the Encoder/Decoder. 143 144 Sample usage model: 145 146 // create and configure Handle 147 var ( 148 bh codec.BincHandle 149 mh codec.MsgpackHandle 150 ch codec.CborHandle 151 ) 152 153 mh.MapType = reflect.TypeOf(map[string]interface{}(nil)) 154 155 // configure extensions 156 // e.g. for msgpack, define functions and enable Time support for tag 1 157 // mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt) 158 159 // create and use decoder/encoder 160 var ( 161 r io.Reader 162 w io.Writer 163 b []byte 164 h = &bh // or mh to use msgpack 165 ) 166 167 dec = codec.NewDecoder(r, h) 168 dec = codec.NewDecoderBytes(b, h) 169 err = dec.Decode(&v) 170 171 enc = codec.NewEncoder(w, h) 172 enc = codec.NewEncoderBytes(&b, h) 173 err = enc.Encode(v) 174 175 //RPC Server 176 go func() { 177 for { 178 conn, err := listener.Accept() 179 rpcCodec := codec.GoRpc.ServerCodec(conn, h) 180 //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h) 181 rpc.ServeCodec(rpcCodec) 182 } 183 }() 184 185 //RPC Communication (client side) 186 conn, err = net.Dial("tcp", "localhost:5555") 187 rpcCodec := codec.GoRpc.ClientCodec(conn, h) 188 //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h) 189 client := rpc.NewClientWithCodec(rpcCodec) 190 191 # Running Tests 192 193 To run tests, use the following: 194 195 go test 196 197 To run the full suite of tests, use the following: 198 199 go test -tags alltests -run Suite 200 201 You can run the tag 'codec.safe' to run tests or build in safe mode. e.g. 202 203 go test -tags codec.safe -run Json 204 go test -tags "alltests codec.safe" -run Suite 205 206 Running Benchmarks 207 208 cd bench 209 go test -bench . -benchmem -benchtime 1s 210 211 Please see http://github.com/ugorji/go-codec-bench . 212 213 # Caveats 214 215 Struct fields matching the following are ignored during encoding and decoding 216 - struct tag value set to - 217 - func, complex numbers, unsafe pointers 218 - unexported and not embedded 219 - unexported and embedded and not struct kind 220 - unexported and embedded pointers (from go1.10) 221 222 Every other field in a struct will be encoded/decoded. 223 224 Embedded fields are encoded as if they exist in the top-level struct, 225 with some caveats. See Encode documentation. 226 */ 227 package codec