-
Notifications
You must be signed in to change notification settings - Fork 1.4k
feat(bigquery): job creation mode GA #12225
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Previously, use of the QUERY_PREVIEW_ENABLED environmental variable governed whether this client would attempt to execute queries without creating jobs. Now that this feature is GA, this client has been updated so that it defaults to using optional job mode, and exposes a new ClientOption (WithJobCreationMode) to override this behavior.
// variables. By setting the environment variable QUERY_PREVIEW_ENABLED to the string | ||
// "TRUE", the client will enable preview features, though behavior may still be | ||
// controlled via the bigquery service as well. Currently, the feature(s) in scope | ||
// include: short mode queries (query execution without corresponding job metadata). | ||
func NewClient(ctx context.Context, projectID string, opts ...option.ClientOption) (*Client, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Quick question for you about usage: is the client intended to be a singleton (pretty typical and understandable if so)? Anecdotally, I know that it's already usable this way (despite the context arg) without issue.
From the readme:
First create a bigquery.Client to use throughout your application: [snip]:# (bq-1)
However, I noticed that the client itself takes a context (for me, a bit unusual for something long-lived in an app's dependency graph). My first glance at this PR made me wonder about the new short query optimization optional job creation option being an option of the client (and in turn every query one generates) rather than to the function(s) that perform query execution. The answer is pretty simple if the readme is inaccurate and it is in fact intended to construct new clients on-demand at query time (which I could see, given that it's a wrapper around the HTTP API) .
Not a big deal either way, because currently-using applications with mixed query workloads could just add one more singleton to their dependency graph (an identical construction of NewClient
, but setting this new option) for usage in places where there's an explicit intent/need to execute with optional job creation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@zcross Apologies, missed your comment initially.
While it would be nice if a client could be truly singleton and stateless, in practice it's not really the case. One of the less obvious things going on under the hood is authentication, which, depending on the particular protocol, can be doing things like refreshing credentials in the background.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for reading! By the way, does the minimalJob
(stub?) returned on the fast path actually work for retrieving query statistics? And on the topic, what would be the best way for a caller to distinguish whether the fast path was exercised or not, e.g., through RowIterator.SourceJob
which naturally points to this minimalJob
result?
If the minimal job is strictly there to support the iterator that's returned, and not other access patterns (which I'd understand), I'm wondering if there's a reasonable way to include query metadata (e.g., statistics like bytes processed, if available) for observability by the caller.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Related to above question: the docs suggest that the iterator will have a nil SourceJob
if no job was created, but it doesn't seem like that's the case (instead, source job is set to the minimalJob object that seems synthetic – not a "real" job naturally)
https://cloud.google.com/bigquery/docs/running-queries#optional-job-creation
From the code snippet for Go ^
// The iterator provides information about the query execution.
// Queries that were run in short query mode will not have the source job
// populated.
if it.SourceJob() == nil {
fmt.Fprintf(w, "Query was run in optional job mode. Query ID: %q\n", it.QueryID())
} else {
j := it.SourceJob()
qualifiedJobID := fmt.Sprintf("%s:%s.%s", j.ProjectID(), j.Location(), j.ID())
fmt.Fprintf(w, "Query was run with job state. Job ID: %q, Query ID: %q\n",
qualifiedJobID, it.QueryID())
}
🤖 I have created a release *beep* *boop* --- ## [1.69.0](https://togithub.com/googleapis/google-cloud-go/compare/bigquery/v1.68.0...bigquery/v1.69.0) (2025-05-27) ### Features * **bigquery/analyticshub:** Add support for Analytics Hub & Marketplace Integration ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b)) * **bigquery/analyticshub:** Adding allow_only_metadata_sharing to Listing resource ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b)) * **bigquery/analyticshub:** Adding CommercialInfo message to the Listing and Subscription resources ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b)) * **bigquery/analyticshub:** Adding delete_commercial and revoke_commercial to DeleteListingRequest and RevokeSubscriptionRequest ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b)) * **bigquery/analyticshub:** Adding DestinationDataset to the Subscription resource ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b)) * **bigquery/analyticshub:** Adding routine field to the SharedResource message ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b)) * **bigquery:** Add support for dataset view and update modes ([#12290](https://togithub.com/googleapis/google-cloud-go/issues/12290)) ([7c1f961](https://togithub.com/googleapis/google-cloud-go/commit/7c1f9616b7ea95436582eb3c40c94e6bd9b48610)) * **bigquery:** Job creation mode GA ([#12225](https://togithub.com/googleapis/google-cloud-go/issues/12225)) ([1d8990d](https://togithub.com/googleapis/google-cloud-go/commit/1d8990dbf2563a5fbc96769ac9c6ea4ed06b239e)) --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
Previously, use of the QUERY_PREVIEW_ENABLED environmental variable governed whether this client would attempt to execute queries without creating jobs. Now that this feature is GA, this client has been updated to expose a new ClientOption (
WithDefaultJobCreationMode
) to control this behavior.This library will no longer use QUERY_PREVIEW_ENABLED to control this behavior, and the environment variable is ignored.