BigQuery - Package cloud.google.com/go/bigquery (v1.18.0)

Package bigquery provides a client for the BigQuery service.

The following assumes a basic familiarity with BigQuery concepts. See https://cloud.google.com/bigquery/docs.

See https://godoc.org/cloud.google.com/go for authentication, timeouts, connection pooling and similar aspects of this package.

Creating a Client

To start working with this package, create a client:

ctx := context.Background()
client, err := bigquery.NewClient(ctx, projectID)
if err != nil {
    // TODO: Handle error.
}

Querying

To query existing tables, create a Query and call its Read method:

q := client.Query(`
    SELECT year, SUM(number) as num
    FROM ` + "`bigquery-public-data.usa_names.usa_1910_2013`" + `
    WHERE name = "William"
    GROUP BY year
    ORDER BY year
`)
it, err := q.Read(ctx)
if err != nil {
    // TODO: Handle error.
}

Then iterate through the resulting rows. You can store a row using anything that implements the ValueLoader interface, or with a slice or map of bigquery.Value. A slice is simplest:

for {
    var values []bigquery.Value
    err := it.Next(&values)
    if err == iterator.Done {
        break
    }
    if err != nil {
        // TODO: Handle error.
    }
    fmt.Println(values)
}

You can also use a struct whose exported fields match the query:

type Count struct {
    Year int
    Num  int
}
for {
    var c Count
    err := it.Next(&c)
    if err == iterator.Done {
        break
    }
    if err != nil {
        // TODO: Handle error.
    }
    fmt.Println(c)
}

You can also start the query running and get the results later. Create the query as above, but call Run instead of Read. This returns a Job, which represents an asynchronous operation.

job, err := q.Run(ctx)
if err != nil {
    // TODO: Handle error.
}

Get the job's ID, a printable string. You can save this string to retrieve the results at a later time, even in another process.

jobID := job.ID()
fmt.Printf("The job ID is %s\n", jobID)

To retrieve the job's results from the ID, first look up the Job:

job, err = client.JobFromID(ctx, jobID)
if err != nil {
    // TODO: Handle error.
}

Use the Job.Read method to obtain an iterator, and loop over the rows. Query.Read is just a convenience method that combines Query.Run and Job.Read.

it, err = job.Read(ctx)
if err != nil {
    // TODO: Handle error.
}
// Proceed with iteration as above.

Datasets and Tables

You can refer to datasets in the client's project with the Dataset method, and in other projects with the DatasetInProject method:

myDataset := client.Dataset("my_dataset")
yourDataset := client.DatasetInProject("your-project-id", "your_dataset")

These methods create references to datasets, not the datasets themselves. You can have a dataset reference even if the dataset doesn't exist yet. Use Dataset.Create to create a dataset from a reference:

if err := myDataset.Create(ctx, nil); err != nil {
    // TODO: Handle error.
}

You can refer to tables with Dataset.Table. Like bigquery.Dataset, bigquery.Table is a reference to an object in BigQuery that may or may not exist.

table := myDataset.Table("my_table")

You can create, delete and update the metadata of tables with methods on Table. For instance, you could create a temporary table with:

err = myDataset.Table("temp").Create(ctx, &bigquery.TableMetadata{
    ExpirationTime: time.Now().Add(1*time.Hour)})
if err != nil {
    // TODO: Handle error.
}

We'll see how to create a table with a schema in the next section.

Schemas

There are two ways to construct schemas with this package. You can build a schema by hand, like so:

schema1 := bigquery.Schema{
    {Name: "Name", Required: true, Type: bigquery.StringFieldType},
    {Name: "Grades", Repeated: true, Type: bigquery.IntegerFieldType},
    {Name: "Optional", Required: false, Type: bigquery.IntegerFieldType},
}

Or you can infer the schema from a struct:

type student struct {
    Name   string
    Grades []int
    Optional bigquery.NullInt64
}
schema2, err := bigquery.InferSchema(student{})
if err != nil {
    // TODO: Handle error.
}
// schema1 and schema2 are identical.

Struct inference supports tags like those of the encoding/json package, so you can change names, ignore fields, or mark a field as nullable (non-required). Fields declared as one of the Null types (NullInt64, NullFloat64, NullString, NullBool, NullTimestamp, NullDate, NullTime, NullDateTime, and NullGeography) are automatically inferred as nullable, so the "nullable" tag is only needed for []byte, *big.Rat and pointer-to-struct fields.

type student2 struct {
    Name     string `bigquery:"full_name"`
    Grades   []int
    Secret   string `bigquery:"-"`
    Optional []byte `bigquery:",nullable"
}
schema3, err := bigquery.InferSchema(student2{})
if err != nil {
    // TODO: Handle error.
}
// schema3 has required fields "full_name" and "Grade", and nullable BYTES field "Optional".

Having constructed a schema, you can create a table with it like so:

if err := table.Create(ctx, &bigquery.TableMetadata{Schema: schema1}); err != nil {
    // TODO: Handle error.
}

Copying

You can copy one or more tables to another table. Begin by constructing a Copier describing the copy. Then set any desired copy options, and finally call Run to get a Job:

copier := myDataset.Table("dest").CopierFrom(myDataset.Table("src"))
copier.WriteDisposition = bigquery.WriteTruncate
job, err = copier.Run(ctx)
if err != nil {
    // TODO: Handle error.
}

You can chain the call to Run if you don't want to set options:

job, err = myDataset.Table("dest").CopierFrom(myDataset.Table("src")).Run(ctx)
if err != nil {
    // TODO: Handle error.
}

You can wait for your job to complete:

status, err := job.Wait(ctx)
if err != nil {
    // TODO: Handle error.
}

Job.Wait polls with exponential backoff. You can also poll yourself, if you wish:

for {
    status, err := job.Status(ctx)
    if err != nil {
        // TODO: Handle error.
    }
    if status.Done() {
        if status.Err() != nil {
            log.Fatalf("Job failed with error %v", status.Err())
        }
        break
    }
    time.Sleep(pollInterval)
}

Loading and Uploading

There are two ways to populate a table with this package: load the data from a Google Cloud Storage object, or upload rows directly from your program.

For loading, first create a GCSReference, configuring it if desired. Then make a Loader, optionally configure it as well, and call its Run method.

gcsRef := bigquery.NewGCSReference("gs://my-bucket/my-object")
gcsRef.AllowJaggedRows = true
loader := myDataset.Table("dest").LoaderFrom(gcsRef)
loader.CreateDisposition = bigquery.CreateNever
job, err = loader.Run(ctx)
// Poll the job for completion if desired, as above.

To upload, first define a type that implements the ValueSaver interface, which has a single method named Save. Then create an Inserter, and call its Put method with a slice of values.

u := table.Inserter()
// Item implements the ValueSaver interface.
items := []*Item{
    {Name: "n1", Size: 32.6, Count: 7},
    {Name: "n2", Size: 4, Count: 2},
    {Name: "n3", Size: 101.5, Count: 1},
}
if err := u.Put(ctx, items); err != nil {
    // TODO: Handle error.
}

You can also upload a struct that doesn't implement ValueSaver. Use the StructSaver type to specify the schema and insert ID by hand, or just supply the struct or struct pointer directly and the schema will be inferred:

type Item2 struct {
    Name  string
    Size  float64
    Count int
}
// Item implements the ValueSaver interface.
items2 := []*Item2{
    {Name: "n1", Size: 32.6, Count: 7},
    {Name: "n2", Size: 4, Count: 2},
    {Name: "n3", Size: 101.5, Count: 1},
}
if err := u.Put(ctx, items2); err != nil {
    // TODO: Handle error.
}

BigQuery allows for higher throughput when omitting insertion IDs. To enable this, specify the sentinel NoDedupeID value for the insertion ID when implementing a ValueSaver.

Extracting

If you've been following so far, extracting data from a BigQuery table into a Google Cloud Storage object will feel familiar. First create an Extractor, then optionally configure it, and lastly call its Run method.

extractor := table.ExtractorTo(gcsRef)
extractor.DisableHeader = true
job, err = extractor.Run(ctx)
// Poll the job for completion if desired, as above.

Errors

Errors returned by this client are often of the type googleapi.Error: https://godoc.org/google.golang.org/api/googleapi#Error

These errors can be introspected for more information by type asserting to the richer *googleapi.Error type. For example:

   if e, ok := err.(*googleapi.Error); ok {
          if e.Code = 409 { ... }
    }

In some cases, your client may received unstructured googleapi.Error error responses. In such cases, it is likely that you have exceeded BigQuery request limits, documented at: https://cloud.google.com/bigquery/quotas

Constants

NumericPrecisionDigits, NumericScaleDigits, BigNumericPrecisionDigits, BigNumericScaleDigits

const (
	// NumericPrecisionDigits is the maximum number of digits in a NUMERIC value.
	NumericPrecisionDigits = 38

	// NumericScaleDigits is the maximum number of digits after the decimal point in a NUMERIC value.
	NumericScaleDigits = 9

	// BigNumericPrecisionDigits is the maximum number of full digits in a BIGNUMERIC value.
	BigNumericPrecisionDigits = 76

	// BigNumericScaleDigits is the maximum number of full digits in a BIGNUMERIC value.
	BigNumericScaleDigits = 38
)

NoDedupeID

const NoDedupeID = "NoDedupeID"

NoDedupeID indicates a streaming insert row wants to opt out of best-effort deduplication. It is EXPERIMENTAL and subject to change or removal without notice.

Scope

const (
	// Scope is the Oauth2 scope for the service.
	// For relevant BigQuery scopes, see:
	// https://developers.google.com/identity/protocols/googlescopes#bigqueryv2
	Scope = "https://www.googleapis.com/auth/bigquery"
)

Variables

NeverExpire

var NeverExpire = time.Time{}.Add(-1)

NeverExpire is a sentinel value used to remove a table'e expiration time.

Functions

func BigNumericString

func BigNumericString(r *big.Rat) string

BigNumericString returns a string representing a *big.Rat in a format compatible with BigQuery SQL. It returns a floating point literal with 38 digits after the decimal point.

func CivilDateTimeString

func CivilDateTimeString(dt civil.DateTime) string

CivilDateTimeString returns a string representing a civil.DateTime in a format compatible with BigQuery SQL. It separate the date and time with a space, and formats the time with CivilTimeString.

Use CivilDateTimeString when using civil.DateTime in DML, for example in INSERT statements.

func CivilTimeString

func CivilTimeString(t civil.Time) string

CivilTimeString returns a string representing a civil.Time in a format compatible with BigQuery SQL. It rounds the time to the nearest microsecond and returns a string with six digits of sub-second precision.

Use CivilTimeString when using civil.Time in DML, for example in INSERT statements.

func NumericString

func NumericString(r *big.Rat) string

NumericString returns a string representing a *big.Rat in a format compatible with BigQuery SQL. It returns a floating-point literal with 9 digits after the decimal point.

func Seed

func Seed(s int64)

Seed seeds this package's random number generator, used for generating job and insert IDs. Use Seed to obtain repeatable, deterministic behavior from bigquery clients. Seed should be called before any clients are created.

AccessEntry

type AccessEntry struct {
	Role       AccessRole // The role of the entity
	EntityType EntityType // The type of entity
	Entity     string     // The entity (individual or group) granted access
	View       *Table     // The view granted access (EntityType must be ViewEntity)
	Routine    *Routine   // The routine granted access (only UDF currently supported)
}

An AccessEntry describes the permissions that an entity has on a dataset.

AccessRole

type AccessRole string

AccessRole is the level of access to grant to a dataset.

OwnerRole, ReaderRole, WriterRole

const (
	// OwnerRole is the OWNER AccessRole.
	OwnerRole AccessRole = "OWNER"
	// ReaderRole is the READER AccessRole.
	ReaderRole AccessRole = "READER"
	// WriterRole is the WRITER AccessRole.
	WriterRole AccessRole = "WRITER"
)

BigtableColumn

type BigtableColumn struct {
	// Qualifier of the column. Columns in the parent column family that have this
	// exact qualifier are exposed as . field. The column field name is the
	// same as the column qualifier.
	Qualifier string

	// If the qualifier is not a valid BigQuery field identifier i.e. does not match
	// [a-zA-Z][a-zA-Z0-9_]*, a valid identifier must be provided as the column field
	// name and is used as field name in queries.
	FieldName string

	// If true, only the latest version of values are exposed for this column.
	// See BigtableColumnFamily.OnlyReadLatest.
	OnlyReadLatest bool

	// The encoding of the values when the type is not STRING.
	// See BigtableColumnFamily.Encoding
	Encoding string

	// The type to convert the value in cells of this column.
	// See BigtableColumnFamily.Type
	Type string
}

BigtableColumn describes how BigQuery should access a Bigtable column.

BigtableColumnFamily

type BigtableColumnFamily struct {
	// Identifier of the column family.
	FamilyID string

	// Lists of columns that should be exposed as individual fields as opposed to a
	// list of (column name, value) pairs. All columns whose qualifier matches a
	// qualifier in this list can be accessed as .. Other columns can be accessed as
	// a list through .Column field.
	Columns []*BigtableColumn

	// The encoding of the values when the type is not STRING. Acceptable encoding values are:
	// - TEXT - indicates values are alphanumeric text strings.
	// - BINARY - indicates values are encoded using HBase Bytes.toBytes family of functions.
	// This can be overridden for a specific column by listing that column in 'columns' and
	// specifying an encoding for it.
	Encoding string

	// If true, only the latest version of values are exposed for all columns in this
	// column family. This can be overridden for a specific column by listing that
	// column in 'columns' and specifying a different setting for that column.
	OnlyReadLatest bool

	// The type to convert the value in cells of this
	// column family. The values are expected to be encoded using HBase
	// Bytes.toBytes function when using the BINARY encoding value.
	// Following BigQuery types are allowed (case-sensitive):
	// BYTES STRING INTEGER FLOAT BOOLEAN.
	// The default type is BYTES. This can be overridden for a specific column by
	// listing that column in 'columns' and specifying a type for it.
	Type string
}

BigtableColumnFamily describes how BigQuery should access a Bigtable column family.

BigtableOptions

type BigtableOptions struct {
	// A list of column families to expose in the table schema along with their
	// types. If omitted, all column families are present in the table schema and
	// their values are read as BYTES.
	ColumnFamilies []*BigtableColumnFamily

	// If true, then the column families that are not specified in columnFamilies
	// list are not exposed in the table schema. Otherwise, they are read with BYTES
	// type values. The default is false.
	IgnoreUnspecifiedColumnFamilies bool

	// If true, then the rowkey column families will be read and converted to string.
	// Otherwise they are read with BYTES type values and users need to manually cast
	// them with CAST if necessary. The default is false.
	ReadRowkeyAsString bool
}

BigtableOptions are additional options for Bigtable external data sources.

CSVOptions

type CSVOptions struct {
	// AllowJaggedRows causes missing trailing optional columns to be tolerated
	// when reading CSV data. Missing values are treated as nulls.
	AllowJaggedRows bool

	// AllowQuotedNewlines sets whether quoted data sections containing
	// newlines are allowed when reading CSV data.
	AllowQuotedNewlines bool

	// Encoding is the character encoding of data to be read.
	Encoding Encoding

	// FieldDelimiter is the separator for fields in a CSV file, used when
	// reading or exporting data. The default is ",".
	FieldDelimiter string

	// Quote is the value used to quote data sections in a CSV file. The
	// default quotation character is the double quote ("), which is used if
	// both Quote and ForceZeroQuote are unset.
	// To specify that no character should be interpreted as a quotation
	// character, set ForceZeroQuote to true.
	// Only used when reading data.
	Quote          string
	ForceZeroQuote bool

	// The number of rows at the top of a CSV file that BigQuery will skip when
	// reading data.
	SkipLeadingRows int64
}

CSVOptions are additional options for CSV external data sources.

Client

type Client struct {
	// Location, if set, will be used as the default location for all subsequent
	// dataset creation and job operations. A location specified directly in one of
	// those operations will override this value.
	Location string
	// contains filtered or unexported fields
}

Client may be used to perform BigQuery operations.

func NewClient

func NewClient(ctx context.Context, projectID string, opts ...option.ClientOption) (*Client, error)

NewClient constructs a new Client which can perform BigQuery operations. Operations performed via the client are billed to the specified GCP project.

Example

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	_ = client // TODO: Use client.
}

func (*Client) Close

func (c *Client) Close() error

Close closes any resources held by the client. Close should be called when the client is no longer needed. It need not be called at program exit.

func (*Client) Dataset

func (c *Client) Dataset(id string) *Dataset

Dataset creates a handle to a BigQuery dataset in the client's project.

Example

package main

import (
	"context"
	"fmt"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	ds := client.Dataset("my_dataset")
	fmt.Println(ds)
}

func (*Client) DatasetInProject

func (c *Client) DatasetInProject(projectID, datasetID string) *Dataset

DatasetInProject creates a handle to a BigQuery dataset in the specified project.

Example

package main

import (
	"context"
	"fmt"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	ds := client.DatasetInProject("their-project-id", "their-dataset")
	fmt.Println(ds)
}

func (*Client) Datasets

func (c *Client) Datasets(ctx context.Context) *DatasetIterator

Datasets returns an iterator over the datasets in a project. The Client's project is used by default, but that can be changed by setting ProjectID on the returned iterator before calling Next.

Example

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	it := client.Datasets(ctx)
	_ = it // TODO: iterate using Next or iterator.Pager.
}

func (*Client) DatasetsInProject (deprecated)

func (c *Client) DatasetsInProject(ctx context.Context, projectID string) *DatasetIterator

DatasetsInProject returns an iterator over the datasets in the provided project.

Deprecated: call Client.Datasets, then set ProjectID on the returned iterator.

Example

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	it := client.DatasetsInProject(ctx, "their-project-id")
	_ = it // TODO: iterate using Next or iterator.Pager.
}

func (*Client) JobFromID

func (c *Client) JobFromID(ctx context.Context, id string) (*Job, error)

JobFromID creates a Job which refers to an existing BigQuery job. The job need not have been created by this package. For example, the job may have been created in the BigQuery console.

For jobs whose location is other than "US" or "EU", set Client.Location or use JobFromIDLocation.

Example

package main

import (
	"context"
	"fmt"

	"cloud.google.com/go/bigquery"
)

func getJobID() string { return "" }

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	jobID := getJobID() // Get a job ID using Job.ID, the console or elsewhere.
	job, err := client.JobFromID(ctx, jobID)
	if err != nil {
		// TODO: Handle error.
	}
	fmt.Println(job.LastStatus()) // Display the job's status.
}

func (*Client) JobFromIDLocation

func (c *Client) JobFromIDLocation(ctx context.Context, id, location string) (j *Job, err error)

JobFromIDLocation creates a Job which refers to an existing BigQuery job. The job need not have been created by this package (for example, it may have been created in the BigQuery console), but it must exist in the specified location.

func (*Client) Jobs

func (c *Client) Jobs(ctx context.Context) *JobIterator

Jobs lists jobs within a project.

Example

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	it := client.Jobs(ctx)
	it.State = bigquery.Running // list only running jobs.
	_ = it                      // TODO: iterate using Next or iterator.Pager.
}

func (*Client) Query

func (c *Client) Query(q string) *Query

Query creates a query with string q. The returned Query may optionally be further configured before its Run method is called.

Examples

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	q := client.Query("select name, num from t1")
	q.DefaultProjectID = "project-id"
	// TODO: set other options on the Query.
	// TODO: Call Query.Run or Query.Read.
}

encryptionKey

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	q := client.Query("select name, num from t1")
	// TODO: Replace this key with a key you have created in Cloud KMS.
	keyName := "projects/P/locations/L/keyRings/R/cryptoKeys/K"
	q.DestinationEncryptionConfig = &bigquery.EncryptionConfig{KMSKeyName: keyName}
	// TODO: set other options on the Query.
	// TODO: Call Query.Run or Query.Read.
}

parameters

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	q := client.Query("select num from t1 where name = @user")
	q.Parameters = []bigquery.QueryParameter{
		{Name: "user", Value: "Elizabeth"},
	}
	// TODO: set other options on the Query.
	// TODO: Call Query.Run or Query.Read.
}

Clustering

type Clustering struct {
	Fields []string
}

Clustering governs the organization of data within a managed table. For more information, see https://cloud.google.com/bigquery/docs/clustered-tables

Compression

type Compression string

Compression is the type of compression to apply when writing data to Google Cloud Storage.

None, Gzip, Deflate, Snappy

const (
	// None specifies no compression.
	None Compression = "NONE"
	// Gzip specifies gzip compression.
	Gzip Compression = "GZIP"
	// Deflate specifies DEFLATE compression for Avro files.
	Deflate Compression = "DEFLATE"
	// Snappy specifies SNAPPY compression for Avro files.
	Snappy Compression = "SNAPPY"
)

Copier

type Copier struct {
	JobIDConfig
	CopyConfig
	// contains filtered or unexported fields
}

A Copier copies data into a BigQuery table from one or more BigQuery tables.

func (*Copier) Run

func (c *Copier) Run(ctx context.Context) (*Job, error)

Run initiates a copy job.

CopyConfig

type CopyConfig struct {
	// Srcs are the tables from which data will be copied.
	Srcs []*Table

	// Dst is the table into which the data will be copied.
	Dst *Table

	// CreateDisposition specifies the circumstances under which the destination table will be created.
	// The default is CreateIfNeeded.
	CreateDisposition TableCreateDisposition

	// WriteDisposition specifies how existing data in the destination table is treated.
	// The default is WriteEmpty.
	WriteDisposition TableWriteDisposition

	// The labels associated with this job.
	Labels map[string]string

	// Custom encryption configuration (e.g., Cloud KMS keys).
	DestinationEncryptionConfig *EncryptionConfig
}

CopyConfig holds the configuration for a copy job.

DataFormat

type DataFormat string

DataFormat describes the format of BigQuery table data.

CSV, Avro, JSON, DatastoreBackup, GoogleSheets, Bigtable, Parquet, ORC, TFSavedModel, XGBoostBooster

const (
	CSV             DataFormat = "CSV"
	Avro            DataFormat = "AVRO"
	JSON            DataFormat = "NEWLINE_DELIMITED_JSON"
	DatastoreBackup DataFormat = "DATASTORE_BACKUP"
	GoogleSheets    DataFormat = "GOOGLE_SHEETS"
	Bigtable        DataFormat = "BIGTABLE"
	Parquet         DataFormat = "PARQUET"
	ORC             DataFormat = "ORC"
	// For BQ ML Models, TensorFlow Saved Model format.
	TFSavedModel DataFormat = "ML_TF_SAVED_MODEL"
	// For BQ ML Models, xgBoost Booster format.
	XGBoostBooster DataFormat = "ML_XGBOOST_BOOSTER"
)

Constants describing the format of BigQuery table data.

Dataset

type Dataset struct {
	ProjectID string
	DatasetID string
	// contains filtered or unexported fields
}

Dataset is a reference to a BigQuery dataset.

func (*Dataset) Create

func (d *Dataset) Create(ctx context.Context, md *DatasetMetadata) (err error)

Create creates a dataset in the BigQuery service. An error will be returned if the dataset already exists. Pass in a DatasetMetadata value to configure the dataset.

Example

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	ds := client.Dataset("my_dataset")
	if err := ds.Create(ctx, &bigquery.DatasetMetadata{Location: "EU"}); err != nil {
		// TODO: Handle error.
	}
}

func (*Dataset) Delete

func (d *Dataset) Delete(ctx context.Context) (err error)

Delete deletes the dataset. Delete will fail if the dataset is not empty.

Example

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	if err := client.Dataset("my_dataset").Delete(ctx); err != nil {
		// TODO: Handle error.
	}
}

func (*Dataset) DeleteWithContents

func (d *Dataset) DeleteWithContents(ctx context.Context) (err error)

DeleteWithContents deletes the dataset, as well as contained resources.

func (*Dataset) Metadata

func (d *Dataset) Metadata(ctx context.Context) (md *DatasetMetadata, err error)

Metadata fetches the metadata for the dataset.

Example

package main

import (
	"context"
	"fmt"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	md, err := client.Dataset("my_dataset").Metadata(ctx)
	if err != nil {
		// TODO: Handle error.
	}
	fmt.Println(md)
}

func (*Dataset) Model

func (d *Dataset) Model(modelID string) *Model

Model creates a handle to a BigQuery model in the dataset. To determine if a model exists, call Model.Metadata. If the model does not already exist, you can create it via execution of a CREATE MODEL query.

func (*Dataset) Models

func (d *Dataset) Models(ctx context.Context) *ModelIterator

Models returns an iterator over the models in the Dataset.

func (*Dataset) Routine

func (d *Dataset) Routine(routineID string) *Routine

Routine creates a handle to a BigQuery routine in the dataset. To determine if a routine exists, call Routine.Metadata.

func (*Dataset) Routines

func (d *Dataset) Routines(ctx context.Context) *RoutineIterator

Routines returns an iterator over the routines in the Dataset.

func (*Dataset) Table

func (d *Dataset) Table(tableID string) *Table

Table creates a handle to a BigQuery table in the dataset. To determine if a table exists, call Table.Metadata. If the table does not already exist, use Table.Create to create it.

Example

package main

import (
	"context"
	"fmt"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	// Table creates a reference to the table. It does not create the actual
	// table in BigQuery; to do so, use Table.Create.
	t := client.Dataset("my_dataset").Table("my_table")
	fmt.Println(t)
}

func (*Dataset) Tables

func (d *Dataset) Tables(ctx context.Context) *TableIterator

Tables returns an iterator over the tables in the Dataset.

Example

package main

import (
	"context"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	it := client.Dataset("my_dataset").Tables(ctx)
	_ = it // TODO: iterate using Next or iterator.Pager.
}

func (*Dataset) Update

func (d *Dataset) Update(ctx context.Context, dm DatasetMetadataToUpdate, etag string) (md *DatasetMetadata, err error)

Update modifies specific Dataset metadata fields. To perform a read-modify-write that protects against intervening reads, set the etag argument to the DatasetMetadata.ETag field from the read. Pass the empty string for etag for a "blind write" that will always succeed.

Examples

blindWrite

package main

import (
	"context"
	"fmt"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	md, err := client.Dataset("my_dataset").Update(ctx, bigquery.DatasetMetadataToUpdate{Name: "blind"}, "")
	if err != nil {
		// TODO: Handle error.
	}
	fmt.Println(md)
}

readModifyWrite

package main

import (
	"context"
	"fmt"

	"cloud.google.com/go/bigquery"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	ds := client.Dataset("my_dataset")
	md, err := ds.Metadata(ctx)
	if err != nil {
		// TODO: Handle error.
	}
	md2, err := ds.Update(ctx,
		bigquery.DatasetMetadataToUpdate{Name: "new " + md.Name},
		md.ETag)
	if err != nil {
		// TODO: Handle error.
	}
	fmt.Println(md2)
}

DatasetIterator

type DatasetIterator struct {
	// ListHidden causes hidden datasets to be listed when set to true.
	// Set before the first call to Next.
	ListHidden bool

	// Filter restricts the datasets returned by label. The filter syntax is described in
	// https://cloud.google.com/bigquery/docs/labeling-datasets#filtering_datasets_using_labels
	// Set before the first call to Next.
	Filter string

	// The project ID of the listed datasets.
	// Set before the first call to Next.
	ProjectID string
	// contains filtered or unexported fields
}

DatasetIterator iterates over the datasets in a project.

func (*DatasetIterator) Next

func (it *DatasetIterator) Next() (*Dataset, error)

Next returns the next Dataset. Its second return value is iterator.Done if there are no more results. Once Next returns Done, all subsequent calls will return Done.

Example

package main

import (
	"context"
	"fmt"

	"cloud.google.com/go/bigquery"
	"google.golang.org/api/iterator"
)

func main() {
	ctx := context.Background()
	client, err := bigquery.NewClient(ctx, "project-id")
	if err != nil {
		// TODO: Handle error.
	}
	it := client.Datasets(ctx)
	for {
		ds, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			// TODO: Handle error.
		}
		fmt.Println(ds)
	}
}

func (*DatasetIterator) PageInfo

func (it *DatasetIterator) PageInfo() *iterator.PageInfo

PageInfo supports pagination. See the google.golang.org/api/iterator package for details.

DatasetMetadata

type DatasetMetadata struct {
	// These fields can be set when creating a dataset.
	Name                    string            // The user-friendly name for this dataset.
	Description             string            // The user-friendly description of this dataset.
	Location                string            // The geo location of the dataset.
	DefaultTableExpiration  time.Duration     // The default expiration time for new tables.
	Labels                  map[string]string // User-provided labels.
	Access                  []*AccessEntry    // Access permissions.
	DefaultEncryptionConfig *EncryptionConfig

	// These fields are read-only.
	CreationTime     time.Time
	LastModifiedTime time.Time // When the dataset or any of its tables were modified.
	FullID           string    // The full dataset ID in the form projectID:datasetID.

	// ETag is the ETag obtained when reading metadata. Pass it to Dataset.Update to
	// ensure that the metadata hasn't changed since it was read.
	ETag string
}

DatasetMetadata contains information about a BigQuery dataset.

DatasetMetadataToUpdate

type DatasetMetadataToUpdate struct {
	Description optional.String // The user-friendly description of this table.
	Name        optional.String // The user-friendly name for this dataset.

	// DefaultTableExpiration is the default expiration time for new tables.
	// If set to time.Duration(0), new tables never expire.
	DefaultTableExpiration optional.Duration

	// DefaultEncryptionConfig defines CMEK settings for new resources created
	// in the dataset.
	DefaultEncryptionConfig *EncryptionConfig

	// The entire access list. It is not possible to replace individual entries.
	Access []*AccessEntry
	// contains filtered or unexported fields
}

DatasetMetadataToUpdate is used when updating a dataset's metadata. Only non-nil fields will be updated.

func (*DatasetMetadataToUpdate) DeleteLabel

func (u *DatasetMetadataToUpdate) DeleteLabel(name string)

DeleteLabel causes a label to be deleted on a call to Update.

func (*DatasetMetadataToUpdate) SetLabel

func (u *DatasetMetadataToUpdate) SetLabel(name, value string)

SetLabel causes a label to be added or modified on a call to Update.

Encoding

type Encoding string

Encoding specifies the character encoding of data to be loaded into BigQuery. See https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.encoding for more details about how this is used.

UTF_8, ISO_8859_1

const (
	// UTF_8 specifies the UTF-8 encoding type.
	UTF_8 Encoding = "UTF-8"
	// ISO_8859_1 specifies the ISO-8859-1 encoding type.
	ISO_8859_1 Encoding = "ISO-8859-1"
)

EncryptionConfig

type EncryptionConfig struct {
	// Describes the Cloud KMS encryption key that will be used to protect
	// destination BigQuery table. The BigQuery Service Account associated with your
	// project requires access to this encryption key.
	KMSKeyName string
}

EncryptionConfig configures customer-managed encryption on tables and ML models.

EntityType

type EntityType int

EntityType is the type of entity in an AccessEntry.

DomainEntity, GroupEmailEntity, UserEmailEntity, SpecialGroupEntity, ViewEntity, IAMMemberEntity, RoutineEntity

const (
	// DomainEntity is a domain (e.g. "example.com").
	DomainEntity EntityType = iota + 1

	// GroupEmailEntity is an email address of a Google Group.
	GroupEmailEntity

	// UserEmailEntity is an email address of an individual user.
	UserEmailEntity

	// SpecialGroupEntity is a special group: one of projectOwners, projectReaders, projectWriters or
	// allAuthenticatedUsers.
	SpecialGroupEntity

	// ViewEntity is a BigQuery logical view.
	ViewEntity

	// IAMMemberEntity represents entities present in IAM but not represented using
	// the other entity types.
	IAMMemberEntity

	// RoutineEntity is a BigQuery routine, referencing a User Defined Function (UDF).
	RoutineEntity
)

Error

type Error struct {
	// Mirrors bq.ErrorProto, but drops DebugInfo
	Location, Message, Reason string
}

An Error contains detailed information about a failed bigquery operation. Detailed description of possible Reasons can be found here: https://cloud.google.com/bigquery/troubleshooting-errors.

func (Error) Error

func (e Error) Error() string

ExplainQueryStage

type ExplainQueryStage struct {
	// CompletedParallelInputs: Number of parallel input segments completed.
	CompletedParallelInputs int64

	// ComputeAvg: Duration the average shard spent on CPU-bound tasks.
	ComputeAvg time.Duration

	// ComputeMax: Duration the slowest shard spent on CPU-bound tasks.
	ComputeMax time.Duration

	// Relative amount of the total time the average shard spent on CPU-bound tasks.
	ComputeRatioAvg float64

	// Relative amount of the total time the slowest shard spent on CPU-bound tasks.
	ComputeRatioMax float64

	// EndTime: Stage end time.
	EndTime time.Time

	// Unique ID for stage within plan.
	ID int64

	// InputStages: IDs for stages that are inputs to this stage.
	InputStages []int64

	// Human-readable name for stage.
	Name string

	// ParallelInputs: Number of parallel input segments to be processed.
	ParallelInputs int64

	// ReadAvg: Duration the average shard spent reading input.
	ReadAvg time.Duration

	// ReadMax: Duration the slowest shard spent reading input.
	ReadMax time.Duration

	// Relative amount of the total time the average shard spent reading input.
	ReadRatioAvg float64

	// Relative amount of the total time the slowest shard spent reading input.
	ReadRatioMax float64

	// Number of records read into the stage.
	RecordsRead int64

	// Number of records written by the stage.
	RecordsWritten int64

	// ShuffleOutputBytes: Total number of bytes written to shuffle.
	ShuffleOutputBytes int64

	// ShuffleOutputBytesSpilled: Total number of bytes written to shuffle
	// and spilled to disk.
	ShuffleOutputBytesSpilled int64

	// StartTime: Stage start time.
	StartTime time.Time

	// Current status for the stage.
	Status string

	// List of operations within the stage in dependency order (approximately
	// chronological).
	Steps []*ExplainQueryStep

	// WaitAvg: Duration the average shard spent waiting to be scheduled.
	WaitAvg time.Duration

	// WaitMax: Duration the slowest shard spent waiting to be scheduled.
	WaitMax time.Duration

	// Relative amount of the total time the average shard spent waiting to be scheduled.
	WaitRatioAvg float64

	// Relative amount of the total time the slowest shard spent waiting to be scheduled.
	WaitRatioMax float64

	// WriteAvg: Duration the average shard spent on writing output.
	WriteAvg time.Duration

	// WriteMax: Duration the slowest shard spent on writing output.
	WriteMax time.Duration

	// Relative amount of the total time the average shard spent on writing output.
	WriteRatioAvg float64

	// Relative amount of the total time the slowest shard spent on writing output.
	WriteRatioMax float64
}

ExplainQueryStage describes one stage of a query.

ExplainQueryStep

type ExplainQueryStep struct {
	// Machine-readable operation type.
	Kind string

	// Human-readable stage descriptions.
	Substeps []string
}

ExplainQueryStep describes one step of a query stage.

ExternalData

type ExternalData interface {
	// contains filtered or unexported methods
}

ExternalData is a table which is stored outside of BigQuery. It is implemented by *ExternalDataConfig. GCSReference also implements it, for backwards compatibility.

ExternalDataConfig

type ExternalDataConfig struct {
	// The format of the data. Required.
	SourceFormat DataFormat

	// The fully-qualified URIs that point to your
	// data in Google Cloud. Required.
	//
	// For Google Cloud Storage URIs, each URI can contain one '*' wildcard character
	// and it must come after the 'bucket' name. Size limits related to load jobs
	// apply to external data sources.
	//
	// For Google Cloud Bigtable URIs, exactly one URI can be specified and it has be
	// a fully specified and valid HTTPS URL for a Google Cloud Bigtable table.
	//
	// For Google Cloud Datastore backups, exactly one URI can be specified. Also,
	// the '*' wildcard character is not allowed.
	SourceURIs []string

	// The schema of the data. Required for CSV and JSON; disallowed for the
	// other formats.
	Schema Schema

	// Try to detect schema and format options automatically.
	// Any option specified explicitly will be honored.
	AutoDetect bool

	// The compression type of the data.
	Compression Compression

	// IgnoreUnknownValues causes values not matching the schema to be
	// tolerated. Unknown values are ignored. For CSV this ignores extra values
	// at the end of a line. For JSON this ignores named values that do not
	// match any column name. If this field is not set, records containing
	// unknown values are treated as bad records. The MaxBadRecords field can
	// be used to customize how bad records are handled.
	IgnoreUnknownValues bool

	// MaxBadRecords is the maximum number of bad records that will be ignored
	// when reading data.
	MaxBadRecords int64

	// Additional options for CSV, GoogleSheets, Bigtable, and Parquet formats.
	Options ExternalDataConfigOptions

	// HivePartitioningOptions allows use of Hive partitioning based on the
	// layout of objects in Google Cloud Storage.
	HivePartitioningOptions *HivePartitioningOptions
}

ExternalDataConfig describes data external to BigQuery that can be used in queries and to create external tables.

ExternalDataConfigOptions

type ExternalDataConfigOptions interface {
	// contains filtered or unexported methods
}

ExternalDataConfigOptions are additional options for external data configurations. This interface is implemented by CSVOptions, GoogleSheetsOptions and BigtableOptions.

ExtractConfig

type ExtractConfig struct {
	// Src is the table from which data will be extracted.
	// Only one of Src or SrcModel should be specified.
	Src *Table

	// SrcModel is the ML model from which the data will be extracted.
	// Only one of Src or SrcModel should be specified.
	SrcModel *Model

	// Dst is the destination into which the data will be extracted.
	Dst *GCSReference

	// DisableHeader disables the printing of a header row in exported data.
	DisableHeader bool

	// The labels associated with this job.
	Labels map[string]string

	// For Avro-based extracts, controls whether logical type annotations are generated.
	//
	// Example:  With this enabled, writing a BigQuery TIMESTAMP column will result in
	// an integer column annotated with the appropriate timestamp-micros/millis annotation
	// in the resulting Avro files.
	UseAvroLogicalTypes bool
}

ExtractConfig holds the configuration for an extract job.

ExtractStatistics

type ExtractStatistics struct {
	// The number of files per destination URI or URI pattern specified in the
	// extract configuration. These values will be in the same order as the
	// URIs specified in the 'destinationUris' field.
	DestinationURIFileCounts []int64
}

ExtractStatistics contains statistics about an extract job.

Extractor

type Extractor struct {
	JobIDConfig
	ExtractConfig
	// contains filtered or unexported fields
}

An Extractor extracts data from a BigQuery table into Google Cloud Storage.

func (*Extractor) Run

func (e *Extractor) Run(ctx context.Context) (j *Job, err error)

Run initiates an extract job.

FieldSchema

type FieldSchema struct {
	// The field name.
	// Must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_),
	// and must start with a letter or underscore.
	// The maximum length is 128 characters.
	Name string

	// A description of the field. The maximum length is 16,384 characters.
	Description string

	// Whether the field may contain multiple values.
	Repeated bool
	// Whether the field is required.  Ignored if Repeated is true.
	Required bool

	// The field data type.  If Type is Record, then this field contains a nested schema,
	// which is described by Schema.
	Type FieldType

	// Annotations for enforcing column-level security constraints.
	PolicyTags *PolicyTagList

	// Describes the nested schema if Type is set to Record.
	Schema Schema
}

FieldSchema describes a single field.

FieldType

type FieldType string

FieldType is the type of field.

StringFieldType, BytesFieldType, IntegerFieldType, FloatFieldType, BooleanFieldType, TimestampFieldType, RecordFieldType, DateFieldType, TimeFieldType, DateTimeFieldType, NumericFieldType, GeographyFieldType, BigNumericFieldType

const (
	// StringFieldType is a string field type.
	StringFieldType FieldType = "STRING"
	// BytesFieldType is a bytes field type.
	BytesFieldType FieldType = "BYTES"
	// IntegerFieldType is a integer field type.
	IntegerFieldType FieldType = "INTEGER"
	// FloatFieldType is a float field type.
	FloatFieldType FieldType = "FLOAT"
	// BooleanFieldType is a boolean field type.
	BooleanFieldType FieldType = "BOOLEAN"
	// TimestampFieldType is a timestamp field type.
	TimestampFieldType FieldType = "TIMESTAMP"
	// RecordFieldType is a record field type. It is typically used to create columns with repeated or nested data.
	RecordFieldType FieldType = "RECORD"
	// DateFieldType is a date field type.
	DateFieldType FieldType = "DATE"
	// TimeFieldType is a time field type.
	TimeFieldType FieldType = "TIME"
	// DateTimeFieldType is a datetime field type.
	DateTimeFieldType FieldType = "DATETIME"
	// NumericFieldType is a numeric field type. Numeric types include integer types, floating point types and the
	// NUMERIC data type.
	NumericFieldType

BigQuery - Package cloud.google.com/go/bigquery (v1.18.0) Stay organized with collections Save and categorize content based on your preferences.

Creating a Client

Querying

Datasets and Tables

Schemas

Copying

Loading and Uploading

Extracting

Errors

Constants

NumericPrecisionDigits, NumericScaleDigits, BigNumericPrecisionDigits, BigNumericScaleDigits

NoDedupeID

Scope

Variables

NeverExpire

Functions

func BigNumericString

func CivilDateTimeString

func CivilTimeString

func NumericString

func Seed

AccessEntry

AccessRole

OwnerRole, ReaderRole, WriterRole

BigtableColumn

BigtableColumnFamily

BigtableOptions

CSVOptions

Client

func NewClient

Example

func (*Client) Close

func (*Client) Dataset

Example

func (*Client) DatasetInProject

Example

func (*Client) Datasets

Example

func (*Client) DatasetsInProject (deprecated)

Example

func (*Client) JobFromID

Example

func (*Client) JobFromIDLocation

func (*Client) Jobs

Example

func (*Client) Query

Examples

encryptionKey

parameters

Clustering

Compression

None, Gzip, Deflate, Snappy

Copier

func (*Copier) Run

CopyConfig

DataFormat

CSV, Avro, JSON, DatastoreBackup, GoogleSheets, Bigtable, Parquet, ORC, TFSavedModel, XGBoostBooster

Dataset

func (*Dataset) Create

Example

func (*Dataset) Delete

Example

func (*Dataset) DeleteWithContents

func (*Dataset) Metadata

Example

func (*Dataset) Model

func (*Dataset) Models

func (*Dataset) Routine

func (*Dataset) Routines

func (*Dataset) Table

Example

func (*Dataset) Tables

Example

func (*Dataset) Update

Examples

blindWrite

readModifyWrite

DatasetIterator

func (*DatasetIterator) Next

Example

BigQuery - Package cloud.google.com/go/bigquery (v1.18.0)