Cloud Datastore - Package cloud.google.com/go/datastore (v1.20.0)

Package datastore provides a client for Google Cloud Datastore.

See https://godoc.org/cloud.google.com/go for authentication, timeouts, connection pooling and similar aspects of this package.

Basic Operations

Entities are the unit of storage and are associated with a key. A key consists of an optional parent key, a string application ID, a string kind (also known as an entity type), and either a StringID or an IntID. A StringID is also known as an entity name or key name.

It is valid to create a key with a zero StringID and a zero IntID; this is called an incomplete key, and does not refer to any saved entity. Putting an entity into the datastore under an incomplete key will cause a unique key to be generated for that entity, with a non-zero IntID.

An entity's contents are a mapping from case-sensitive field names to values. Valid value types are:

• Signed integers (int, int8, int16, int32 and int64)
• bool
• string
• float32 and float64
• []byte (up to 1 megabyte in length)
• Any type whose underlying type is one of the above predeclared types
• *Key
• GeoPoint
• time.Time (stored with microsecond precision, retrieved as UTC)
• Structs whose fields are all valid value types
• Pointers to structs whose fields are all valid value types
• Slices of any of the above
• Pointers to a signed integer, bool, string, float32, or float64

Slices of structs are valid, as are structs that contain slices.

The Get and Put functions load and save an entity's contents. An entity's contents are typically represented by a struct pointer.

Example code:

type Entity struct {
    Value string
}

func main() {
    ctx := context.Background()

    // Create a datastore client. In a typical application, you would create
    // a single client which is reused for every datastore operation.
    dsClient, err := datastore.NewClient(ctx, "my-project")
    if err != nil {
        // Handle error.
    }
    defer dsClient.Close()

    k := datastore.NameKey("Entity", "stringID", nil)
    e := new(Entity)
    if err := dsClient.Get(ctx, k, e); err != nil {
        // Handle error.
    }

    old := e.Value
    e.Value = "Hello World!"

    if _, err := dsClient.Put(ctx, k, e); err != nil {
        // Handle error.
    }

    fmt.Printf("Updated value from %q to %q\n", old, e.Value)
}

GetMulti, PutMulti and DeleteMulti are batch versions of the Get, Put and Delete functions. They take a []*Key instead of a *Key, and may return a datastore.MultiError when encountering partial failure.

Mutate generalizes PutMulti and DeleteMulti to a sequence of any Datastore mutations. It takes a series of mutations created with NewInsert, NewUpdate, NewUpsert and NewDelete and applies them. Datastore.Mutate uses non-transactional mode; if atomicity is required, use Transaction.Mutate instead.

Properties

An entity's contents can be represented by a variety of types. These are typically struct pointers, but can also be any type that implements the PropertyLoadSaver interface. If using a struct pointer, you do not have to explicitly implement the PropertyLoadSaver interface; the datastore will automatically convert via reflection. If a struct pointer does implement PropertyLoadSaver then those methods will be used in preference to the default behavior for struct pointers. Struct pointers are more strongly typed and are easier to use; PropertyLoadSavers are more flexible.

The actual types passed do not have to match between Get and Put calls or even across different calls to datastore. It is valid to put a *PropertyList and get that same entity as a *myStruct, or put a *myStruct0 and get a *myStruct1. Conceptually, any entity is saved as a sequence of properties, and is loaded into the destination value on a property-by-property basis. When loading into a struct pointer, an entity that cannot be completely represented (such as a missing field) will result in an ErrFieldMismatch error but it is up to the caller whether this error is fatal, recoverable or ignorable.

By default, for struct pointers, all properties are potentially indexed, and the property name is the same as the field name (and hence must start with an upper case letter).

Fields may have a datastore:"name,options" tag. The tag name is the property name, which must be one or more valid Go identifiers joined by ".", but may start with a lower case letter. An empty tag name means to just use the field name. A "-" tag name means that the datastore will ignore that field.

The only valid options are "omitempty", "noindex" and "flatten".

If the options include "omitempty" and the value of the field is an empty value, then the field will be omitted on Save. Empty values are defined as false, 0, a nil pointer, a nil interface value, the zero time.Time, and any empty slice or string. (Empty slices are never saved, even without "omitempty".) Other structs, including GeoPoint, are never considered empty.

If options include "noindex" then the field will not be indexed. All fields are indexed by default. Strings or byte slices longer than 1500 bytes cannot be indexed; fields used to store long strings and byte slices must be tagged with "noindex" or they will cause Put operations to fail.

For a nested struct field, the options may also include "flatten". This indicates that the immediate fields and any nested substruct fields of the nested struct should be flattened. See below for examples.

To use multiple options together, separate them by a comma. The order does not matter.

If the options is "" then the comma may be omitted.

Example code:

// A and B are renamed to a and b.
// A, C and J are not indexed.
// D's tag is equivalent to having no tag at all (E).
// I is ignored entirely by the datastore.
// J has tag information for both the datastore and json packages.
type TaggedStruct struct {
    A int `datastore:"a,noindex"`
    B int `datastore:"b"`
    C int `datastore:",noindex"`
    D int `datastore:""`
    E int
    I int `datastore:"-"`
    J int `datastore:",noindex" json:"j"`
}

Slice Fields

A field of slice type corresponds to a Datastore array property, except for []byte, which corresponds to a Datastore blob.

Zero-length slice fields are not saved. Slice fields of length 1 or greater are saved as Datastore arrays. When a zero-length Datastore array is loaded into a slice field, the slice field remains unchanged.

If a non-array value is loaded into a slice field, the result will be a slice with one element, containing the value.

Loading Nulls

Loading a Datastore Null into a basic type (int, float, etc.) results in a zero value. Loading a Null into a slice of basic type results in a slice of size 1 containing the zero value. Loading a Null into a pointer field results in nil. Loading a Null into a field of struct type is an error.

Pointer Fields

A struct field can be a pointer to a signed integer, floating-point number, string or bool. Putting a non-nil pointer will store its dereferenced value. Putting a nil pointer will store a Datastore Null property, unless the field is marked omitempty, in which case no property will be stored.

Loading a Null into a pointer field sets the pointer to nil. Loading any other value allocates new storage with the value, and sets the field to point to it.

Key Field

If the struct contains a *datastore.Key field tagged with the name "key", its value will be ignored on Put. When reading the Entity back into the Go struct, the field will be populated with the *datastore.Key value used to query for the Entity.

Example code:

type MyEntity struct {
    A int
    K *datastore.Key `datastore:"__key__"`
}

func main() {
    ctx := context.Background()
    dsClient, err := datastore.NewClient(ctx, "my-project")
    if err != nil {
        // Handle error.
    }
    defer dsClient.Close()

    k := datastore.NameKey("Entity", "stringID", nil)
    e := MyEntity{A: 12}
    if _, err := dsClient.Put(ctx, k, &e); err != nil {
        // Handle error.
    }

    var entities []MyEntity
    q := datastore.NewQuery("Entity").FilterField("A", "=", 12).Limit(1)
    if _, err := dsClient.GetAll(ctx, q, &entities); err != nil {
        // Handle error
    }

    log.Println(entities[0])
    // Prints {12 /Entity,stringID}
}

Structured Properties

If the struct pointed to contains other structs, then the nested or embedded structs are themselves saved as Entity values. For example, given these definitions:

type Inner struct {
    W int32
    X string
}

type Outer struct {
    I Inner
}

then an Outer would have one property, Inner, encoded as an Entity value.

Note: embedded struct fields must be named to be encoded as an Entity. For example, in case of a type Outer with an embedded field Inner:

type Outer struct {
    Inner
}

all the Inner struct fields will be treated as fields of Outer itself.

If an outer struct is tagged "noindex" then all of its implicit flattened fields are effectively "noindex".

If the Inner struct contains a *Key field with the name "key", like so:

type Inner struct {
    W int32
    X string
    K *datastore.Key `datastore:"__key__"`
}

type Outer struct {
    I Inner
}

then the value of K will be used as the Key for Inner, represented as an Entity value in datastore.

If any nested struct fields should be flattened, instead of encoded as Entity values, the nested struct field should be tagged with the "flatten" option. For example, given the following:

type Inner1 struct {
    W int32
    X string
}

type Inner2 struct {
    Y float64
}

type Inner3 struct {
    Z bool
}

type Inner4 struct {
    WW int
}

type Inner5 struct {
    X Inner4
}

type Outer struct {
    A int16
    I []Inner1 `datastore:",flatten"`
    J Inner2   `datastore:",flatten"`
    K Inner5   `datastore:",flatten"`
    Inner3     `datastore:",flatten"`
}

an Outer's properties would be equivalent to those of:

type OuterEquivalent struct {
    A          int16
    IDotW      []int32  `datastore:"I.W"`
    IDotX      []string `datastore:"I.X"`
    JDotY      float64  `datastore:"J.Y"`
    KDotXDotWW int      `datastore:"K.X.WW"`
    Z          bool
}

Note that the "flatten" option cannot be used for Entity value fields or PropertyLoadSaver implementers. The server will reject any dotted field names for an Entity value.

The PropertyLoadSaver Interface

An entity's contents can also be represented by any type that implements the PropertyLoadSaver interface. This type may be a struct pointer, but it does not have to be. The datastore package will call Load when getting the entity's contents, and Save when putting the entity's contents. Possible uses include deriving non-stored fields, verifying fields, or indexing a field only if its value is positive.

Example code:

type CustomPropsExample struct {
    I, J int
    // Sum is not stored, but should always be equal to I + J.
    Sum int `datastore:"-"`
}

func (x *CustomPropsExample) Load(ps []datastore.Property) error {
    // Load I and J as usual.
    if err := datastore.LoadStruct(x, ps); err != nil {
        return err
    }
    // Derive the Sum field.
    x.Sum = x.I + x.J
    return nil
}

func (x *CustomPropsExample) Save() ([]datastore.Property, error) {
    // Validate the Sum field.
    if x.Sum != x.I + x.J {
        return nil, errors.New("CustomPropsExample has inconsistent sum")
    }
    // Save I and J as usual. The code below is equivalent to calling
    // "return datastore.SaveStruct(x)", but is done manually for
    // demonstration purposes.
    return []datastore.Property{
        {
            Name:  "I",
            Value: int64(x.I),
        },
        {
            Name:  "J",
            Value: int64(x.J),
        },
    }, nil
}

The *PropertyList type implements PropertyLoadSaver, and can therefore hold an arbitrary entity's contents.

The KeyLoader Interface

If a type implements the PropertyLoadSaver interface, it may also want to implement the KeyLoader interface. The KeyLoader interface exists to allow implementations of PropertyLoadSaver to also load an Entity's Key into the Go type. This type may be a struct pointer, but it does not have to be. The datastore package will call LoadKey when getting the entity's contents, after calling Load.

Example code:

type WithKeyExample struct {
    I int
    Key   *datastore.Key
}

func (x *WithKeyExample) LoadKey(k *datastore.Key) error {
    x.Key = k
    return nil
}

func (x *WithKeyExample) Load(ps []datastore.Property) error {
    // Load I as usual.
    return datastore.LoadStruct(x, ps)
}

func (x *WithKeyExample) Save() ([]datastore.Property, error) {
    // Save I as usual.
    return datastore.SaveStruct(x)
}

To load a Key into a struct which does not implement the PropertyLoadSaver interface, see the "Key Field" section above.

Queries

Queries retrieve entities based on their properties or key's ancestry. Running a query yields an iterator of results: either keys or (key, entity) pairs. Queries are re-usable and it is safe to call Query.Run from concurrent goroutines. Iterators are not safe for concurrent use.

Queries are immutable, and are either created by calling NewQuery, or derived from an existing query by calling a method like Filter or Order that returns a new query value. A query is typically constructed by calling NewQuery followed by a chain of zero or more such methods. These methods are:

• Ancestor and Filter constrain the entities returned by running a query.
• Order affects the order in which they are returned.
• Project constrains the fields returned.
• Distinct de-duplicates projected entities.
• KeysOnly makes the iterator return only keys, not (key, entity) pairs.
• Start, End, Offset and Limit define which sub-sequence of matching entities to return. Start and End take cursors, Offset and Limit take integers. Start and Offset affect the first result, End and Limit affect the last result. If both Start and Offset are set, then the offset is relative to Start. If both End and Limit are set, then the earliest constraint wins. Limit is relative to Start+Offset, not relative to End. As a special case, a negative limit means unlimited.

Example code:

type Widget struct {
    Description string
    Price       int
}

func printWidgets(ctx context.Context, client *datastore.Client) {
    q := datastore.NewQuery("Widget").
        FilterField("Price", "<", 1000).
        Order("-Price")

    t := client.Run(ctx, q)
    for {
        var x Widget
        key, err := t.Next(&x)
        if err == iterator.Done {
            break
        }
        if err != nil {
            // Handle error.
        }
        fmt.Printf("Key=%v\nWidget=%#v\n\n", key, x)
    }
}

Transactions

Client.RunInTransaction runs a function in a transaction.

Example code:

type Counter struct {
    Count int
}

func incCount(ctx context.Context, client *datastore.Client) {
    var count int
    key := datastore.NameKey("Counter", "singleton", nil)
    _, err := client.RunInTransaction(ctx, func(tx *datastore.Transaction) error {
        var x Counter
        if err := tx.Get(key, &x); err != nil && err != datastore.ErrNoSuchEntity {
            return err
        }
        x.Count++
        if _, err := tx.Put(key, &x); err != nil {
            return err
        }
        count = x.Count
        return nil
    })
    if err != nil {
        // Handle error.
    }
    // The value of count is only valid once the transaction is successful
    // (RunInTransaction has returned nil).
    fmt.Printf("Count=%d\n", count)
}

Pass the ReadOnly option to RunInTransaction if your transaction is used only for Get, GetMulti or queries. Read-only transactions are more efficient.

Google Cloud Datastore Emulator

This package supports the Cloud Datastore emulator, which is useful for testing and development. Environment variables are used to indicate that datastore traffic should be directed to the emulator instead of the production Datastore service.

To install and set up the emulator and its environment variables, see the documentation at https://cloud.google.com/datastore/docs/tools/datastore-emulator.

To use the emulator with this library, you can set the DATASTORE_EMULATOR_HOST environment variable to the address at which your emulator is running. This will send requests to that address instead of to Cloud Datastore. You can then create and use a client as usual:

// Set DATASTORE_EMULATOR_HOST environment variable.
err := os.Setenv("DATASTORE_EMULATOR_HOST", "localhost:9000")
if err != nil {
    // TODO: Handle error.
}
// Create client as usual.
client, err := datastore.NewClient(ctx, "my-project-id")
if err != nil {
    // TODO: Handle error.
}
defer client.Close()

DefaultDatabaseID is ID of the default database denoted by an empty string

DetectProjectID is a sentinel value that instructs NewClient to detect the project ID. It is given in place of the projectID argument. NewClient will use the project ID from the given credentials or the default credentials (https://developers.google.com/accounts/docs/application-default-credentials) if no credentials were provided. When providing credentials, not all options will allow NewClient to extract the project ID. Specifically a JWT does not have the project ID encoded.

ScopeDatastore grants permissions to view and/or manage datastore entities

ErrConcurrentTransaction is returned when a transaction is rolled back due to a conflict with a concurrent transaction.

LoadStruct loads the properties from p to dst. dst must be a struct pointer.

The values of dst's unmatched struct fields are not modified, and matching slice-typed fields are not reset before appending to them. In particular, it is recommended to pass a pointer to a zero valued struct on each LoadStruct call.

WithIgnoreFieldMismatch allows ignoring ErrFieldMismatch error while reading or querying data. WARNING: Ignoring ErrFieldMismatch can cause data loss while writing back to Datastore. E.g. if entity written to Datastore is {X: 1, Y:2} and it is read into type NewStruct struct{X int}, then {X:1} is returned. Now, if this is written back to Datastore, there will be no Y field left for this entity in Datastore

AggregationQuery allows for generating aggregation results of an underlying basic query. A single AggregationQuery can contain multiple aggregations.

WithAvg specifies that the aggregation query should provide an average of the values of the provided field in the results returned by the underlying Query. The alias argument can be empty or a valid Datastore entity property name. It can be used as key in the AggregationResult to get the sum value. If alias is empty, Datastore will autogenerate a key.

WithCount specifies that the aggregation query provide a count of results returned by the underlying Query.

WithSum specifies that the aggregation query should provide a sum of the values of the provided field in the results returned by the underlying Query. The alias argument can be empty or a valid Datastore entity property name. It can be used as key in the AggregationResult to get the sum value. If alias is empty, Datastore will autogenerate a key.

AggregationResult contains the results of an aggregation query.

AggregationWithOptionsResult contains the results of an aggregation query run with options.

AndFilter represents the intersection of two or more filters.

Client is a client for reading and writing data in a datastore dataset.

NewClient creates a new Client for a given dataset. If the project ID is empty, it is derived from the DATASTORE_PROJECT_ID environment variable. If the DATASTORE_EMULATOR_HOST environment variable is set, client will use its value to connect to a locally-running datastore emulator. DetectProjectID can be passed as the projectID argument to instruct NewClient to detect the project ID from the credentials. Call (*Client).Close() when done with the client.

NewClientWithDatabase creates a new Client for given dataset and database. If the project ID is empty, it is derived from the DATASTORE_PROJECT_ID environment variable. If the DATASTORE_EMULATOR_HOST environment variable is set, client will use its value to connect to a locally-running datastore emulator. DetectProjectID can be passed as the projectID argument to instruct NewClientWithDatabase to detect the project ID from the credentials. Call (*Client).Close() when done with the client.

AllocateIDs accepts a slice of incomplete keys and returns a slice of complete keys that are guaranteed to be valid in the datastore.

Close closes the Client. Call Close to clean up resources when done with the Client.

Count returns the number of results for the given query.

The running time and number of API calls made by Count scale linearly with the sum of the query's offset and limit. Unless the result count is expected to be small, it is best to specify a limit; otherwise Count will continue until it finishes counting or the provided context expires.

Deprecated. Use Client.RunAggregationQuery() instead.

Delete deletes the entity for the given key.

DeleteMulti is a batch version of Delete.

err may be a MultiError. See ExampleMultiError to check it.

Get loads the entity stored for key into dst, which must be a struct pointer or implement PropertyLoadSaver. If there is no such entity for the key, Get returns ErrNoSuchEntity.

The values of dst's unmatched struct fields are not modified, and matching slice-typed fields are not reset before appending to them. In particular, it is recommended to pass a pointer to a zero valued struct on each Get call.

ErrFieldMismatch is returned when a field is to be loaded into a different type than the one it was stored from, or when a field is missing or unexported in the destination struct. ErrFieldMismatch is only returned if dst is a struct pointer.

GetAll runs the provided query in the given context and returns all keys that match that query, as well as appending the values to dst.

dst must have type []S or *[]S or *[]P, for some struct type S or some non- interface, non-pointer type P such that P or *P implements PropertyLoadSaver.

As a special case, *PropertyList is an invalid type for dst, even though a PropertyList is a slice of structs. It is treated as invalid to avoid being mistakenly passed when *[]PropertyList was intended.

The keys returned by GetAll will be in a 1-1 correspondence with the entities added to dst.

If q is a “keys-only” query, GetAll ignores dst and only returns the keys.

The running time and number of API calls made by GetAll scale linearly with with the sum of the query's offset and limit. Unless the result count is expected to be small, it is best to specify a limit; otherwise GetAll will continue until it finishes collecting results or the provided context expires.

GetAllWithOptions is similar to GetAll but runs the query with provided options

GetMulti is a batch version of Get.

dst must be a []S, []*S, []I or []P, for some struct type S, some interface type I, or some non-interface non-pointer type P such that P or *P implements PropertyLoadSaver. If an []I, each element must be a valid dst for Get: it must be a struct pointer or implement PropertyLoadSaver.

As a special case, PropertyList is an invalid type for dst, even though a PropertyList is a slice of structs. It is treated as invalid to avoid being mistakenly passed when []PropertyList was intended.

err may be a MultiError. See ExampleMultiError to check it.

Mutate applies one or more mutations. Mutations are applied in non-transactional mode. If you need atomicity, use Transaction.Mutate. It returns the keys of the argument Mutations, in the same order.

If any of the mutations are invalid, Mutate returns a MultiError with the errors. Mutate returns a MultiError in this case even if there is only one Mutation. See ExampleMultiError to check it.

NewTransaction starts a new transaction.

Put saves the entity src into the datastore with the given key. src must be a struct pointer or implement PropertyLoadSaver; if the struct pointer has any unexported fields they will be skipped. If the key is incomplete, the returned key will be a unique key generated by the datastore.

PutMulti is a batch version of Put.

src must satisfy the same conditions as the dst argument to GetMulti. err may be a MultiError. See ExampleMultiError to check it.