gtsocial-umbx

Unnamed repository; edit this file 'description' to name the repository.
Log | Files | Refs | README | LICENSE

README.md (10203B)


      1 # Package Documentation for github.com/ugorji/go/codec
      2 
      3 Package codec provides a High Performance, Feature-Rich Idiomatic Go 1.4+
      4 codec/encoding library for binc, msgpack, cbor, json.
      5 
      6 Supported Serialization formats are:
      7 
      8   - msgpack: https://github.com/msgpack/msgpack
      9   - binc: http://github.com/ugorji/binc
     10   - cbor: http://cbor.io http://tools.ietf.org/html/rfc7049
     11   - json: http://json.org http://tools.ietf.org/html/rfc7159
     12   - simple:
     13 
     14 This package will carefully use 'package unsafe' for performance reasons
     15 in specific places. You can build without unsafe use by passing the safe or
     16 appengine tag i.e. 'go install -tags=codec.safe ...'.
     17 
     18 This library works with both the standard `gc` and the `gccgo` compilers.
     19 
     20 For detailed usage information, read the primer at
     21 http://ugorji.net/blog/go-codec-primer .
     22 
     23 The idiomatic Go support is as seen in other encoding packages in the standard
     24 library (ie json, xml, gob, etc).
     25 
     26 Rich Feature Set includes:
     27 
     28   - Simple but extremely powerful and feature-rich API
     29   - Support for go 1.4 and above, while selectively using newer APIs for later
     30     releases
     31   - Excellent code coverage ( > 90% )
     32   - Very High Performance. Our extensive benchmarks show us outperforming Gob,
     33     Json, Bson, etc by 2-4X.
     34   - Careful selected use of 'unsafe' for targeted performance gains.
     35   - 100% safe mode supported, where 'unsafe' is not used at all.
     36   - Lock-free (sans mutex) concurrency for scaling to 100's of cores
     37   - In-place updates during decode, with option to zero value in maps and slices
     38     prior to decode
     39   - Coerce types where appropriate e.g. decode an int in the stream into a
     40     float, decode numbers from formatted strings, etc
     41   - Corner Cases: Overflows, nil maps/slices, nil values in streams are handled
     42     correctly
     43   - Standard field renaming via tags
     44   - Support for omitting empty fields during an encoding
     45   - Encoding from any value and decoding into pointer to any value (struct,
     46     slice, map, primitives, pointers, interface{}, etc)
     47   - Extensions to support efficient encoding/decoding of any named types
     48   - Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
     49   - Support using existence of `IsZero() bool` to determine if a value is a zero
     50     value. Analogous to time.Time.IsZero() bool.
     51   - Decoding without a schema (into a interface{}). Includes Options to
     52     configure what specific map or slice type to use when decoding an encoded
     53     list or map into a nil interface{}
     54   - Mapping a non-interface type to an interface, so we can decode appropriately
     55     into any interface type with a correctly configured non-interface value.
     56   - Encode a struct as an array, and decode struct from an array in the data
     57     stream
     58   - Option to encode struct keys as numbers (instead of strings) (to support
     59     structured streams with fields encoded as numeric codes)
     60   - Comprehensive support for anonymous fields
     61   - Fast (no-reflection) encoding/decoding of common maps and slices
     62   - Code-generation for faster performance, supported in go 1.6+
     63   - Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
     64   - Support indefinite-length formats to enable true streaming (for formats
     65     which support it e.g. json, cbor)
     66   - Support canonical encoding, where a value is ALWAYS encoded as same
     67     sequence of bytes. This mostly applies to maps, where iteration order is
     68     non-deterministic.
     69   - NIL in data stream decoded as zero value
     70   - Never silently skip data when decoding. User decides whether to return an
     71     error or silently skip data when keys or indexes in the data stream do not
     72     map to fields in the struct.
     73   - Detect and error when encoding a cyclic reference (instead of stack overflow
     74     shutdown)
     75   - Encode/Decode from/to chan types (for iterative streaming support)
     76   - Drop-in replacement for encoding/json. `json:` key in struct tag supported.
     77   - Provides a RPC Server and Client Codec for net/rpc communication protocol.
     78   - Handle unique idiosyncrasies of codecs e.g. For messagepack,
     79     configure how ambiguities in handling raw bytes are resolved and provide
     80     rpc server/client codec to support msgpack-rpc protocol defined at:
     81     https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
     82 
     83 # Extension Support
     84 
     85 Users can register a function to handle the encoding or decoding of their custom
     86 types.
     87 
     88 There are no restrictions on what the custom type can be. Some examples:
     89 
     90 ```go
     91     type BisSet   []int
     92     type BitSet64 uint64
     93     type UUID     string
     94     type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
     95     type GifImage struct { ... }
     96 ```
     97 
     98 As an illustration, MyStructWithUnexportedFields would normally be encoded as
     99 an empty map because it has no exported fields, while UUID would be encoded as a
    100 string. However, with extension support, you can encode any of these however you
    101 like.
    102 
    103 There is also seamless support provided for registering an extension (with a
    104 tag) but letting the encoding mechanism default to the standard way.
    105 
    106 # Custom Encoding and Decoding
    107 
    108 This package maintains symmetry in the encoding and decoding halfs. We determine
    109 how to encode or decode by walking this decision tree
    110 
    111   - is there an extension registered for the type?
    112   - is type a codec.Selfer?
    113   - is format binary, and is type a encoding.BinaryMarshaler and
    114     BinaryUnmarshaler?
    115   - is format specifically json, and is type a encoding/json.Marshaler and
    116     Unmarshaler?
    117   - is format text-based, and type an encoding.TextMarshaler and
    118     TextUnmarshaler?
    119   - else we use a pair of functions based on the "kind" of the type e.g. map,
    120     slice, int64, etc
    121 
    122 This symmetry is important to reduce chances of issues happening because the
    123 encoding and decoding sides are out of sync e.g. decoded via very specific
    124 encoding.TextUnmarshaler but encoded via kind-specific generalized mode.
    125 
    126 Consequently, if a type only defines one-half of the symmetry (e.g.
    127 it implements UnmarshalJSON() but not MarshalJSON() ), then that type doesn't
    128 satisfy the check and we will continue walking down the decision tree.
    129 
    130 # RPC
    131 
    132 RPC Client and Server Codecs are implemented, so the codecs can be used with the
    133 standard net/rpc package.
    134 
    135 # Usage
    136 
    137 The Handle is SAFE for concurrent READ, but NOT SAFE for concurrent
    138 modification.
    139 
    140 The Encoder and Decoder are NOT safe for concurrent use.
    141 
    142 Consequently, the usage model is basically:
    143 
    144   - Create and initialize the Handle before any use. Once created, DO NOT modify
    145     it.
    146   - Multiple Encoders or Decoders can now use the Handle concurrently. They only
    147     read information off the Handle (never write).
    148   - However, each Encoder or Decoder MUST not be used concurrently
    149   - To re-use an Encoder/Decoder, call Reset(...) on it first. This allows you
    150     use state maintained on the Encoder/Decoder.
    151 
    152 Sample usage model:
    153 
    154 ```go
    155     // create and configure Handle
    156     var (
    157       bh codec.BincHandle
    158       mh codec.MsgpackHandle
    159       ch codec.CborHandle
    160     )
    161 
    162     mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
    163 
    164     // configure extensions
    165     // e.g. for msgpack, define functions and enable Time support for tag 1
    166     // mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)
    167 
    168     // create and use decoder/encoder
    169     var (
    170       r io.Reader
    171       w io.Writer
    172       b []byte
    173       h = &bh // or mh to use msgpack
    174     )
    175 
    176     dec = codec.NewDecoder(r, h)
    177     dec = codec.NewDecoderBytes(b, h)
    178     err = dec.Decode(&v)
    179 
    180     enc = codec.NewEncoder(w, h)
    181     enc = codec.NewEncoderBytes(&b, h)
    182     err = enc.Encode(v)
    183 
    184     //RPC Server
    185     go func() {
    186         for {
    187             conn, err := listener.Accept()
    188             rpcCodec := codec.GoRpc.ServerCodec(conn, h)
    189             //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
    190             rpc.ServeCodec(rpcCodec)
    191         }
    192     }()
    193 
    194     //RPC Communication (client side)
    195     conn, err = net.Dial("tcp", "localhost:5555")
    196     rpcCodec := codec.GoRpc.ClientCodec(conn, h)
    197     //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
    198     client := rpc.NewClientWithCodec(rpcCodec)
    199 ```
    200 
    201 # Running Tests
    202 
    203 To run tests, use the following:
    204 
    205 ```
    206     go test
    207 ```
    208 
    209 To run the full suite of tests, use the following:
    210 
    211 ```
    212     go test -tags alltests -run Suite
    213 ```
    214 
    215 You can run the tag 'codec.safe' to run tests or build in safe mode. e.g.
    216 
    217 ```
    218     go test -tags codec.safe -run Json
    219     go test -tags "alltests codec.safe" -run Suite
    220 ```
    221 
    222 # Running Benchmarks
    223 
    224 ```
    225     cd bench
    226     go test -bench . -benchmem -benchtime 1s
    227 ```
    228 
    229 Please see http://github.com/ugorji/go-codec-bench .
    230 
    231 # Caveats
    232 
    233 Struct fields matching the following are ignored during encoding and decoding
    234 
    235   - struct tag value set to -
    236   - func, complex numbers, unsafe pointers
    237   - unexported and not embedded
    238   - unexported and embedded and not struct kind
    239   - unexported and embedded pointers (from go1.10)
    240 
    241 Every other field in a struct will be encoded/decoded.
    242 
    243 Embedded fields are encoded as if they exist in the top-level struct, with some
    244 caveats. See Encode documentation.
    245 
    246 ## Exported Package API
    247 
    248 ```go
    249 const CborStreamBytes byte = 0x5f ...
    250 const GenVersion = 28
    251 var SelfExt = &extFailWrapper{}
    252 var GoRpc goRpc
    253 var MsgpackSpecRpc msgpackSpecRpc
    254 func GenHelper() (g genHelper)
    255 type BasicHandle struct{ ... }
    256 type BincHandle struct{ ... }
    257 type BytesExt interface{ ... }
    258 type CborHandle struct{ ... }
    259 type DecodeOptions struct{ ... }
    260 type Decoder struct{ ... }
    261     func NewDecoder(r io.Reader, h Handle) *Decoder
    262     func NewDecoderBytes(in []byte, h Handle) *Decoder
    263     func NewDecoderString(s string, h Handle) *Decoder
    264 type EncodeOptions struct{ ... }
    265 type Encoder struct{ ... }
    266     func NewEncoder(w io.Writer, h Handle) *Encoder
    267     func NewEncoderBytes(out *[]byte, h Handle) *Encoder
    268 type Ext interface{ ... }
    269 type Handle interface{ ... }
    270 type InterfaceExt interface{ ... }
    271 type JsonHandle struct{ ... }
    272 type MapBySlice interface{ ... }
    273 type MissingFielder interface{ ... }
    274 type MsgpackHandle struct{ ... }
    275 type MsgpackSpecRpcMultiArgs []interface{}
    276 type RPCOptions struct{ ... }
    277 type Raw []byte
    278 type RawExt struct{ ... }
    279 type Rpc interface{ ... }
    280 type Selfer interface{ ... }
    281 type SimpleHandle struct{ ... }
    282 type TypeInfos struct{ ... }
    283     func NewTypeInfos(tags []string) *TypeInfos
    284 ```