README.md (10203B)
1 # Package Documentation for github.com/ugorji/go/codec 2 3 Package codec provides a High Performance, Feature-Rich Idiomatic Go 1.4+ 4 codec/encoding library for binc, msgpack, cbor, json. 5 6 Supported Serialization formats are: 7 8 - msgpack: https://github.com/msgpack/msgpack 9 - binc: http://github.com/ugorji/binc 10 - cbor: http://cbor.io http://tools.ietf.org/html/rfc7049 11 - json: http://json.org http://tools.ietf.org/html/rfc7159 12 - simple: 13 14 This package will carefully use 'package unsafe' for performance reasons 15 in specific places. You can build without unsafe use by passing the safe or 16 appengine tag i.e. 'go install -tags=codec.safe ...'. 17 18 This library works with both the standard `gc` and the `gccgo` compilers. 19 20 For detailed usage information, read the primer at 21 http://ugorji.net/blog/go-codec-primer . 22 23 The idiomatic Go support is as seen in other encoding packages in the standard 24 library (ie json, xml, gob, etc). 25 26 Rich Feature Set includes: 27 28 - Simple but extremely powerful and feature-rich API 29 - Support for go 1.4 and above, while selectively using newer APIs for later 30 releases 31 - Excellent code coverage ( > 90% ) 32 - Very High Performance. Our extensive benchmarks show us outperforming Gob, 33 Json, Bson, etc by 2-4X. 34 - Careful selected use of 'unsafe' for targeted performance gains. 35 - 100% safe mode supported, where 'unsafe' is not used at all. 36 - Lock-free (sans mutex) concurrency for scaling to 100's of cores 37 - In-place updates during decode, with option to zero value in maps and slices 38 prior to decode 39 - Coerce types where appropriate e.g. decode an int in the stream into a 40 float, decode numbers from formatted strings, etc 41 - Corner Cases: Overflows, nil maps/slices, nil values in streams are handled 42 correctly 43 - Standard field renaming via tags 44 - Support for omitting empty fields during an encoding 45 - Encoding from any value and decoding into pointer to any value (struct, 46 slice, map, primitives, pointers, interface{}, etc) 47 - Extensions to support efficient encoding/decoding of any named types 48 - Support encoding.(Binary|Text)(M|Unm)arshaler interfaces 49 - Support using existence of `IsZero() bool` to determine if a value is a zero 50 value. Analogous to time.Time.IsZero() bool. 51 - Decoding without a schema (into a interface{}). Includes Options to 52 configure what specific map or slice type to use when decoding an encoded 53 list or map into a nil interface{} 54 - Mapping a non-interface type to an interface, so we can decode appropriately 55 into any interface type with a correctly configured non-interface value. 56 - Encode a struct as an array, and decode struct from an array in the data 57 stream 58 - Option to encode struct keys as numbers (instead of strings) (to support 59 structured streams with fields encoded as numeric codes) 60 - Comprehensive support for anonymous fields 61 - Fast (no-reflection) encoding/decoding of common maps and slices 62 - Code-generation for faster performance, supported in go 1.6+ 63 - Support binary (e.g. messagepack, cbor) and text (e.g. json) formats 64 - Support indefinite-length formats to enable true streaming (for formats 65 which support it e.g. json, cbor) 66 - Support canonical encoding, where a value is ALWAYS encoded as same 67 sequence of bytes. This mostly applies to maps, where iteration order is 68 non-deterministic. 69 - NIL in data stream decoded as zero value 70 - Never silently skip data when decoding. User decides whether to return an 71 error or silently skip data when keys or indexes in the data stream do not 72 map to fields in the struct. 73 - Detect and error when encoding a cyclic reference (instead of stack overflow 74 shutdown) 75 - Encode/Decode from/to chan types (for iterative streaming support) 76 - Drop-in replacement for encoding/json. `json:` key in struct tag supported. 77 - Provides a RPC Server and Client Codec for net/rpc communication protocol. 78 - Handle unique idiosyncrasies of codecs e.g. For messagepack, 79 configure how ambiguities in handling raw bytes are resolved and provide 80 rpc server/client codec to support msgpack-rpc protocol defined at: 81 https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md 82 83 # Extension Support 84 85 Users can register a function to handle the encoding or decoding of their custom 86 types. 87 88 There are no restrictions on what the custom type can be. Some examples: 89 90 ```go 91 type BisSet []int 92 type BitSet64 uint64 93 type UUID string 94 type MyStructWithUnexportedFields struct { a int; b bool; c []int; } 95 type GifImage struct { ... } 96 ``` 97 98 As an illustration, MyStructWithUnexportedFields would normally be encoded as 99 an empty map because it has no exported fields, while UUID would be encoded as a 100 string. However, with extension support, you can encode any of these however you 101 like. 102 103 There is also seamless support provided for registering an extension (with a 104 tag) but letting the encoding mechanism default to the standard way. 105 106 # Custom Encoding and Decoding 107 108 This package maintains symmetry in the encoding and decoding halfs. We determine 109 how to encode or decode by walking this decision tree 110 111 - is there an extension registered for the type? 112 - is type a codec.Selfer? 113 - is format binary, and is type a encoding.BinaryMarshaler and 114 BinaryUnmarshaler? 115 - is format specifically json, and is type a encoding/json.Marshaler and 116 Unmarshaler? 117 - is format text-based, and type an encoding.TextMarshaler and 118 TextUnmarshaler? 119 - else we use a pair of functions based on the "kind" of the type e.g. map, 120 slice, int64, etc 121 122 This symmetry is important to reduce chances of issues happening because the 123 encoding and decoding sides are out of sync e.g. decoded via very specific 124 encoding.TextUnmarshaler but encoded via kind-specific generalized mode. 125 126 Consequently, if a type only defines one-half of the symmetry (e.g. 127 it implements UnmarshalJSON() but not MarshalJSON() ), then that type doesn't 128 satisfy the check and we will continue walking down the decision tree. 129 130 # RPC 131 132 RPC Client and Server Codecs are implemented, so the codecs can be used with the 133 standard net/rpc package. 134 135 # Usage 136 137 The Handle is SAFE for concurrent READ, but NOT SAFE for concurrent 138 modification. 139 140 The Encoder and Decoder are NOT safe for concurrent use. 141 142 Consequently, the usage model is basically: 143 144 - Create and initialize the Handle before any use. Once created, DO NOT modify 145 it. 146 - Multiple Encoders or Decoders can now use the Handle concurrently. They only 147 read information off the Handle (never write). 148 - However, each Encoder or Decoder MUST not be used concurrently 149 - To re-use an Encoder/Decoder, call Reset(...) on it first. This allows you 150 use state maintained on the Encoder/Decoder. 151 152 Sample usage model: 153 154 ```go 155 // create and configure Handle 156 var ( 157 bh codec.BincHandle 158 mh codec.MsgpackHandle 159 ch codec.CborHandle 160 ) 161 162 mh.MapType = reflect.TypeOf(map[string]interface{}(nil)) 163 164 // configure extensions 165 // e.g. for msgpack, define functions and enable Time support for tag 1 166 // mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt) 167 168 // create and use decoder/encoder 169 var ( 170 r io.Reader 171 w io.Writer 172 b []byte 173 h = &bh // or mh to use msgpack 174 ) 175 176 dec = codec.NewDecoder(r, h) 177 dec = codec.NewDecoderBytes(b, h) 178 err = dec.Decode(&v) 179 180 enc = codec.NewEncoder(w, h) 181 enc = codec.NewEncoderBytes(&b, h) 182 err = enc.Encode(v) 183 184 //RPC Server 185 go func() { 186 for { 187 conn, err := listener.Accept() 188 rpcCodec := codec.GoRpc.ServerCodec(conn, h) 189 //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h) 190 rpc.ServeCodec(rpcCodec) 191 } 192 }() 193 194 //RPC Communication (client side) 195 conn, err = net.Dial("tcp", "localhost:5555") 196 rpcCodec := codec.GoRpc.ClientCodec(conn, h) 197 //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h) 198 client := rpc.NewClientWithCodec(rpcCodec) 199 ``` 200 201 # Running Tests 202 203 To run tests, use the following: 204 205 ``` 206 go test 207 ``` 208 209 To run the full suite of tests, use the following: 210 211 ``` 212 go test -tags alltests -run Suite 213 ``` 214 215 You can run the tag 'codec.safe' to run tests or build in safe mode. e.g. 216 217 ``` 218 go test -tags codec.safe -run Json 219 go test -tags "alltests codec.safe" -run Suite 220 ``` 221 222 # Running Benchmarks 223 224 ``` 225 cd bench 226 go test -bench . -benchmem -benchtime 1s 227 ``` 228 229 Please see http://github.com/ugorji/go-codec-bench . 230 231 # Caveats 232 233 Struct fields matching the following are ignored during encoding and decoding 234 235 - struct tag value set to - 236 - func, complex numbers, unsafe pointers 237 - unexported and not embedded 238 - unexported and embedded and not struct kind 239 - unexported and embedded pointers (from go1.10) 240 241 Every other field in a struct will be encoded/decoded. 242 243 Embedded fields are encoded as if they exist in the top-level struct, with some 244 caveats. See Encode documentation. 245 246 ## Exported Package API 247 248 ```go 249 const CborStreamBytes byte = 0x5f ... 250 const GenVersion = 28 251 var SelfExt = &extFailWrapper{} 252 var GoRpc goRpc 253 var MsgpackSpecRpc msgpackSpecRpc 254 func GenHelper() (g genHelper) 255 type BasicHandle struct{ ... } 256 type BincHandle struct{ ... } 257 type BytesExt interface{ ... } 258 type CborHandle struct{ ... } 259 type DecodeOptions struct{ ... } 260 type Decoder struct{ ... } 261 func NewDecoder(r io.Reader, h Handle) *Decoder 262 func NewDecoderBytes(in []byte, h Handle) *Decoder 263 func NewDecoderString(s string, h Handle) *Decoder 264 type EncodeOptions struct{ ... } 265 type Encoder struct{ ... } 266 func NewEncoder(w io.Writer, h Handle) *Encoder 267 func NewEncoderBytes(out *[]byte, h Handle) *Encoder 268 type Ext interface{ ... } 269 type Handle interface{ ... } 270 type InterfaceExt interface{ ... } 271 type JsonHandle struct{ ... } 272 type MapBySlice interface{ ... } 273 type MissingFielder interface{ ... } 274 type MsgpackHandle struct{ ... } 275 type MsgpackSpecRpcMultiArgs []interface{} 276 type RPCOptions struct{ ... } 277 type Raw []byte 278 type RawExt struct{ ... } 279 type Rpc interface{ ... } 280 type Selfer interface{ ... } 281 type SimpleHandle struct{ ... } 282 type TypeInfos struct{ ... } 283 func NewTypeInfos(tags []string) *TypeInfos 284 ```