Source file src/encoding/gob/doc.go
1 // Copyright 2009 The Go Authors. All rights reserved. 2 // Use of this source code is governed by a BSD-style 3 // license that can be found in the LICENSE file. 4 5 /* 6 Package gob manages streams of gobs - binary values exchanged between an 7 [Encoder] (transmitter) and a [Decoder] (receiver). A typical use is transporting 8 arguments and results of remote procedure calls (RPCs) such as those provided by 9 [net/rpc]. 10 11 The implementation compiles a custom codec for each data type in the stream and 12 is most efficient when a single [Encoder] is used to transmit a stream of values, 13 amortizing the cost of compilation. 14 15 # Basics 16 17 A stream of gobs is self-describing. Each data item in the stream is preceded by 18 a specification of its type, expressed in terms of a small set of predefined 19 types. Pointers are not transmitted, but the things they point to are 20 transmitted; that is, the values are flattened. Nil pointers are not permitted, 21 as they have no value. Recursive types work fine, but 22 recursive values (data with cycles) are problematic. This may change. 23 24 To use gobs, create an [Encoder] and present it with a series of data items as 25 values or addresses that can be dereferenced to values. The [Encoder] makes sure 26 all type information is sent before it is needed. At the receive side, a 27 [Decoder] retrieves values from the encoded stream and unpacks them into local 28 variables. 29 30 # Types and Values 31 32 The source and destination values/types need not correspond exactly. For structs, 33 fields (identified by name) that are in the source but absent from the receiving 34 variable will be ignored. Fields that are in the receiving variable but missing 35 from the transmitted type or value will be ignored in the destination. If a field 36 with the same name is present in both, their types must be compatible. Both the 37 receiver and transmitter will do all necessary indirection and dereferencing to 38 convert between gobs and actual Go values. For instance, a gob type that is 39 schematically, 40 41 struct { A, B int } 42 43 can be sent from or received into any of these Go types: 44 45 struct { A, B int } // the same 46 *struct { A, B int } // extra indirection of the struct 47 struct { *A, **B int } // extra indirection of the fields 48 struct { A, B int64 } // different concrete value type; see below 49 50 It may also be received into any of these: 51 52 struct { A, B int } // the same 53 struct { B, A int } // ordering doesn't matter; matching is by name 54 struct { A, B, C int } // extra field (C) ignored 55 struct { B int } // missing field (A) ignored; data will be dropped 56 struct { B, C int } // missing field (A) ignored; extra field (C) ignored. 57 58 Attempting to receive into these types will draw a decode error: 59 60 struct { A int; B uint } // change of signedness for B 61 struct { A int; B float } // change of type for B 62 struct { } // no field names in common 63 struct { C, D int } // no field names in common 64 65 Integers are transmitted two ways: arbitrary precision signed integers or 66 arbitrary precision unsigned integers. There is no int8, int16 etc. 67 discrimination in the gob format; there are only signed and unsigned integers. As 68 described below, the transmitter sends the value in a variable-length encoding; 69 the receiver accepts the value and stores it in the destination variable. 70 Floating-point numbers are always sent using IEEE 754 64-bit precision (see 71 below). 72 73 Signed integers may be received into any signed integer variable: int, int16, etc.; 74 unsigned integers may be received into any unsigned integer variable; and floating 75 point values may be received into any floating point variable. However, 76 the destination variable must be able to represent the value or the decode 77 operation will fail. 78 79 Structs, arrays and slices are also supported. Structs encode and decode only 80 exported fields. Strings and arrays of bytes are supported with a special, 81 efficient representation (see below). When a slice is decoded, if the existing 82 slice has capacity the slice will be extended in place; if not, a new array is 83 allocated. Regardless, the length of the resulting slice reports the number of 84 elements decoded. 85 86 In general, if allocation is required, the decoder will allocate memory. If not, 87 it will update the destination variables with values read from the stream. It does 88 not initialize them first, so if the destination is a compound value such as a 89 map, struct, or slice, the decoded values will be merged elementwise into the 90 existing variables. 91 92 Functions and channels will not be sent in a gob. Attempting to encode such a value 93 at the top level will fail. A struct field of chan or func type is treated exactly 94 like an unexported field and is ignored. 95 96 Gob can encode a value of any type implementing the [GobEncoder] or 97 [encoding.BinaryMarshaler] interfaces by calling the corresponding method, 98 in that order of preference. 99 100 Gob can decode a value of any type implementing the [GobDecoder] or 101 [encoding.BinaryUnmarshaler] interfaces by calling the corresponding method, 102 again in that order of preference. 103 104 # Encoding Details 105 106 This section documents the encoding, details that are not important for most 107 users. Details are presented bottom-up. 108 109 An unsigned integer is sent one of two ways. If it is less than 128, it is sent 110 as a byte with that value. Otherwise it is sent as a minimal-length big-endian 111 (high byte first) byte stream holding the value, preceded by one byte holding the 112 byte count, negated. Thus 0 is transmitted as (00), 7 is transmitted as (07) and 113 256 is transmitted as (FE 01 00). 114 115 A boolean is encoded within an unsigned integer: 0 for false, 1 for true. 116 117 A signed integer, i, is encoded within an unsigned integer, u. Within u, bits 1 118 upward contain the value; bit 0 says whether they should be complemented upon 119 receipt. The encode algorithm looks like this: 120 121 var u uint 122 if i < 0 { 123 u = (^uint(i) << 1) | 1 // complement i, bit 0 is 1 124 } else { 125 u = (uint(i) << 1) // do not complement i, bit 0 is 0 126 } 127 encodeUnsigned(u) 128 129 The low bit is therefore analogous to a sign bit, but making it the complement bit 130 instead guarantees that the largest negative integer is not a special case. For 131 example, -129=^128=(^256>>1) encodes as (FE 01 01). 132 133 Floating-point numbers are always sent as a representation of a float64 value. 134 That value is converted to a uint64 using [math.Float64bits]. The uint64 is then 135 byte-reversed and sent as a regular unsigned integer. The byte-reversal means the 136 exponent and high-precision part of the mantissa go first. Since the low bits are 137 often zero, this can save encoding bytes. For instance, 17.0 is encoded in only 138 three bytes (FE 31 40). 139 140 Strings and slices of bytes are sent as an unsigned count followed by that many 141 uninterpreted bytes of the value. 142 143 All other slices and arrays are sent as an unsigned count followed by that many 144 elements using the standard gob encoding for their type, recursively. 145 146 Maps are sent as an unsigned count followed by that many key, element 147 pairs. Empty but non-nil maps are sent, so if the receiver has not allocated 148 one already, one will always be allocated on receipt unless the transmitted map 149 is nil and not at the top level. 150 151 In slices and arrays, as well as maps, all elements, even zero-valued elements, 152 are transmitted, even if all the elements are zero. 153 154 Structs are sent as a sequence of (field number, field value) pairs. The field 155 value is sent using the standard gob encoding for its type, recursively. If a 156 field has the zero value for its type (except for arrays; see above), it is omitted 157 from the transmission. The field number is defined by the type of the encoded 158 struct: the first field of the encoded type is field 0, the second is field 1, 159 etc. When encoding a value, the field numbers are delta encoded for efficiency 160 and the fields are always sent in order of increasing field number; the deltas are 161 therefore unsigned. The initialization for the delta encoding sets the field 162 number to -1, so an unsigned integer field 0 with value 7 is transmitted as unsigned 163 delta = 1, unsigned value = 7 or (01 07). Finally, after all the fields have been 164 sent a terminating mark denotes the end of the struct. That mark is a delta=0 165 value, which has representation (00). 166 167 Interface types are not checked for compatibility; all interface types are 168 treated, for transmission, as members of a single "interface" type, analogous to 169 int or []byte - in effect they're all treated as interface{}. Interface values 170 are transmitted as a string identifying the concrete type being sent (a name 171 that must be pre-defined by calling [Register]), followed by a byte count of the 172 length of the following data (so the value can be skipped if it cannot be 173 stored), followed by the usual encoding of concrete (dynamic) value stored in 174 the interface value. (A nil interface value is identified by the empty string 175 and transmits no value.) Upon receipt, the decoder verifies that the unpacked 176 concrete item satisfies the interface of the receiving variable. 177 178 If a value is passed to [Encoder.Encode] and the type is not a struct (or pointer to struct, 179 etc.), for simplicity of processing it is represented as a struct of one field. 180 The only visible effect of this is to encode a zero byte after the value, just as 181 after the last field of an encoded struct, so that the decode algorithm knows when 182 the top-level value is complete. 183 184 The representation of types is described below. When a type is defined on a given 185 connection between an [Encoder] and [Decoder], it is assigned a signed integer type 186 id. When [Encoder.Encode](v) is called, it makes sure there is an id assigned for 187 the type of v and all its elements and then it sends the pair (typeid, encoded-v) 188 where typeid is the type id of the encoded type of v and encoded-v is the gob 189 encoding of the value v. 190 191 To define a type, the encoder chooses an unused, positive type id and sends the 192 pair (-type id, encoded-type) where encoded-type is the gob encoding of a wireType 193 description, constructed from these types: 194 195 type wireType struct { 196 ArrayT *ArrayType 197 SliceT *SliceType 198 StructT *StructType 199 MapT *MapType 200 GobEncoderT *gobEncoderType 201 BinaryMarshalerT *gobEncoderType 202 TextMarshalerT *gobEncoderType 203 204 } 205 type arrayType struct { 206 CommonType 207 Elem typeId 208 Len int 209 } 210 type CommonType struct { 211 Name string // the name of the struct type 212 Id int // the id of the type, repeated so it's inside the type 213 } 214 type sliceType struct { 215 CommonType 216 Elem typeId 217 } 218 type structType struct { 219 CommonType 220 Field []*fieldType // the fields of the struct. 221 } 222 type fieldType struct { 223 Name string // the name of the field. 224 Id int // the type id of the field, which must be already defined 225 } 226 type mapType struct { 227 CommonType 228 Key typeId 229 Elem typeId 230 } 231 type gobEncoderType struct { 232 CommonType 233 } 234 235 If there are nested type ids, the types for all inner type ids must be defined 236 before the top-level type id is used to describe an encoded-v. 237 238 For simplicity in setup, the connection is defined to understand these types a 239 priori, as well as the basic gob types int, uint, etc. Their ids are: 240 241 bool 1 242 int 2 243 uint 3 244 float 4 245 []byte 5 246 string 6 247 complex 7 248 interface 8 249 // gap for reserved ids. 250 WireType 16 251 ArrayType 17 252 CommonType 18 253 SliceType 19 254 StructType 20 255 FieldType 21 256 // 22 is slice of fieldType. 257 MapType 23 258 259 Finally, each message created by a call to Encode is preceded by an encoded 260 unsigned integer count of the number of bytes remaining in the message. After 261 the initial type name, interface values are wrapped the same way; in effect, the 262 interface value acts like a recursive invocation of Encode. 263 264 In summary, a gob stream looks like 265 266 (byteCount (-type id, encoding of a wireType)* (type id, encoding of a value))* 267 268 where * signifies zero or more repetitions and the type id of a value must 269 be predefined or be defined before the value in the stream. 270 271 Compatibility: Any future changes to the package will endeavor to maintain 272 compatibility with streams encoded using previous versions. That is, any released 273 version of this package should be able to decode data written with any previously 274 released version, subject to issues such as security fixes. See the Go compatibility 275 document for background: https://golang.org/doc/go1compat 276 277 See "Gobs of data" for a design discussion of the gob wire format: 278 https://blog.golang.org/gobs-of-data 279 280 # Security 281 282 This package is not designed to be hardened against adversarial inputs, and is 283 outside the scope of https://go.dev/security/policy. In particular, the [Decoder] 284 does only basic sanity checking on decoded input sizes, and its limits are not 285 configurable. Care should be taken when decoding gob data from untrusted 286 sources, which may consume significant resources. 287 */ 288 package gob 289 290 /* 291 Grammar: 292 293 Tokens starting with a lower case letter are terminals; int(n) 294 and uint(n) represent the signed/unsigned encodings of the value n. 295 296 GobStream: 297 DelimitedMessage* 298 DelimitedMessage: 299 uint(lengthOfMessage) Message 300 Message: 301 TypeSequence TypedValue 302 TypeSequence 303 (TypeDefinition DelimitedTypeDefinition*)? 304 DelimitedTypeDefinition: 305 uint(lengthOfTypeDefinition) TypeDefinition 306 TypedValue: 307 int(typeId) Value 308 TypeDefinition: 309 int(-typeId) encodingOfWireType 310 Value: 311 SingletonValue | StructValue 312 SingletonValue: 313 uint(0) FieldValue 314 FieldValue: 315 builtinValue | ArrayValue | MapValue | SliceValue | StructValue | InterfaceValue 316 InterfaceValue: 317 NilInterfaceValue | NonNilInterfaceValue 318 NilInterfaceValue: 319 uint(0) 320 NonNilInterfaceValue: 321 ConcreteTypeName TypeSequence InterfaceContents 322 ConcreteTypeName: 323 uint(lengthOfName) [already read=n] name 324 InterfaceContents: 325 int(concreteTypeId) DelimitedValue 326 DelimitedValue: 327 uint(length) Value 328 ArrayValue: 329 uint(n) FieldValue*n [n elements] 330 MapValue: 331 uint(n) (FieldValue FieldValue)*n [n (key, value) pairs] 332 SliceValue: 333 uint(n) FieldValue*n [n elements] 334 StructValue: 335 (uint(fieldDelta) FieldValue)* 336 */ 337 338 /* 339 For implementers and the curious, here is an encoded example. Given 340 type Point struct {X, Y int} 341 and the value 342 p := Point{22, 33} 343 the bytes transmitted that encode p will be: 344 1f ff 81 03 01 01 05 50 6f 69 6e 74 01 ff 82 00 345 01 02 01 01 58 01 04 00 01 01 59 01 04 00 00 00 346 07 ff 82 01 2c 01 42 00 347 They are determined as follows. 348 349 Since this is the first transmission of type Point, the type descriptor 350 for Point itself must be sent before the value. This is the first type 351 we've sent on this Encoder, so it has type id 65 (0 through 64 are 352 reserved). 353 354 1f // This item (a type descriptor) is 31 bytes long. 355 ff 81 // The negative of the id for the type we're defining, -65. 356 // This is one byte (indicated by FF = -1) followed by 357 // ^-65<<1 | 1. The low 1 bit signals to complement the 358 // rest upon receipt. 359 360 // Now we send a type descriptor, which is itself a struct (wireType). 361 // The type of wireType itself is known (it's built in, as is the type of 362 // all its components), so we just need to send a *value* of type wireType 363 // that represents type "Point". 364 // Here starts the encoding of that value. 365 // Set the field number implicitly to -1; this is done at the beginning 366 // of every struct, including nested structs. 367 03 // Add 3 to field number; now 2 (wireType.structType; this is a struct). 368 // structType starts with an embedded CommonType, which appears 369 // as a regular structure here too. 370 01 // add 1 to field number (now 0); start of embedded CommonType. 371 01 // add 1 to field number (now 0, the name of the type) 372 05 // string is (unsigned) 5 bytes long 373 50 6f 69 6e 74 // wireType.structType.CommonType.name = "Point" 374 01 // add 1 to field number (now 1, the id of the type) 375 ff 82 // wireType.structType.CommonType._id = 65 376 00 // end of embedded wiretype.structType.CommonType struct 377 01 // add 1 to field number (now 1, the field array in wireType.structType) 378 02 // There are two fields in the type (len(structType.field)) 379 01 // Start of first field structure; add 1 to get field number 0: field[0].name 380 01 // 1 byte 381 58 // structType.field[0].name = "X" 382 01 // Add 1 to get field number 1: field[0].id 383 04 // structType.field[0].typeId is 2 (signed int). 384 00 // End of structType.field[0]; start structType.field[1]; set field number to -1. 385 01 // Add 1 to get field number 0: field[1].name 386 01 // 1 byte 387 59 // structType.field[1].name = "Y" 388 01 // Add 1 to get field number 1: field[1].id 389 04 // struct.Type.field[1].typeId is 2 (signed int). 390 00 // End of structType.field[1]; end of structType.field. 391 00 // end of wireType.structType structure 392 00 // end of wireType structure 393 394 Now we can send the Point value. Again the field number resets to -1: 395 396 07 // this value is 7 bytes long 397 ff 82 // the type number, 65 (1 byte (-FF) followed by 65<<1) 398 01 // add one to field number, yielding field 0 399 2c // encoding of signed "22" (0x2c = 44 = 22<<1); Point.x = 22 400 01 // add one to field number, yielding field 1 401 42 // encoding of signed "33" (0x42 = 66 = 33<<1); Point.y = 33 402 00 // end of structure 403 404 The type encoding is long and fairly intricate but we send it only once. 405 If p is transmitted a second time, the type is already known so the 406 output will be just: 407 408 07 ff 82 01 2c 01 42 00 409 410 A single non-struct value at top level is transmitted like a field with 411 delta tag 0. For instance, a signed integer with value 3 presented as 412 the argument to Encode will emit: 413 414 03 04 00 06 415 416 Which represents: 417 418 03 // this value is 3 bytes long 419 04 // the type number, 2, represents an integer 420 00 // tag delta 0 421 06 // value 3 422 423 */ 424