Skip to content

feat(bigquery): job creation mode GA #12225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 27, 2025
Merged

Conversation

shollyman
Copy link
Contributor

@shollyman shollyman commented May 8, 2025

Previously, use of the QUERY_PREVIEW_ENABLED environmental variable governed whether this client would attempt to execute queries without creating jobs. Now that this feature is GA, this client has been updated to expose a new ClientOption (WithDefaultJobCreationMode) to control this behavior.

This library will no longer use QUERY_PREVIEW_ENABLED to control this behavior, and the environment variable is ignored.

Previously, use of the QUERY_PREVIEW_ENABLED environmental variable
governed whether this client would attempt to execute queries without
creating jobs.  Now that this feature is GA, this client has been
updated so that it defaults to using optional job mode, and exposes
a new ClientOption (WithJobCreationMode) to override this behavior.
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the BigQuery API. label May 8, 2025
@shollyman shollyman requested a review from alvarowolfx May 20, 2025 18:30
@shollyman shollyman marked this pull request as ready for review May 20, 2025 18:30
@shollyman shollyman requested review from a team as code owners May 20, 2025 18:30
@shollyman shollyman added the automerge Merge the pull request once unit tests and other checks pass. label May 27, 2025
@shollyman shollyman merged commit 1d8990d into googleapis:main May 27, 2025
8 checks passed
@shollyman shollyman deleted the jco-ga branch May 27, 2025 16:00
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label May 27, 2025
// variables. By setting the environment variable QUERY_PREVIEW_ENABLED to the string
// "TRUE", the client will enable preview features, though behavior may still be
// controlled via the bigquery service as well. Currently, the feature(s) in scope
// include: short mode queries (query execution without corresponding job metadata).
func NewClient(ctx context.Context, projectID string, opts ...option.ClientOption) (*Client, error) {
Copy link

@zcross zcross May 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick question for you about usage: is the client intended to be a singleton (pretty typical and understandable if so)? Anecdotally, I know that it's already usable this way (despite the context arg) without issue.

From the readme:

First create a bigquery.Client to use throughout your application: [snip]:# (bq-1)

However, I noticed that the client itself takes a context (for me, a bit unusual for something long-lived in an app's dependency graph). My first glance at this PR made me wonder about the new short query optimization optional job creation option being an option of the client (and in turn every query one generates) rather than to the function(s) that perform query execution. The answer is pretty simple if the readme is inaccurate and it is in fact intended to construct new clients on-demand at query time (which I could see, given that it's a wrapper around the HTTP API) .

Not a big deal either way, because currently-using applications with mixed query workloads could just add one more singleton to their dependency graph (an identical construction of NewClient, but setting this new option) for usage in places where there's an explicit intent/need to execute with optional job creation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zcross Apologies, missed your comment initially.

While it would be nice if a client could be truly singleton and stateless, in practice it's not really the case. One of the less obvious things going on under the hood is authentication, which, depending on the particular protocol, can be doing things like refreshing credentials in the background.

Copy link

@zcross zcross May 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reading! By the way, does the minimalJob (stub?) returned on the fast path actually work for retrieving query statistics? And on the topic, what would be the best way for a caller to distinguish whether the fast path was exercised or not, e.g., through RowIterator.SourceJob which naturally points to this minimalJob result?

If the minimal job is strictly there to support the iterator that's returned, and not other access patterns (which I'd understand), I'm wondering if there's a reasonable way to include query metadata (e.g., statistics like bytes processed, if available) for observability by the caller.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to above question: the docs suggest that the iterator will have a nil SourceJob if no job was created, but it doesn't seem like that's the case (instead, source job is set to the minimalJob object that seems synthetic – not a "real" job naturally)

https://cloud.google.com/bigquery/docs/running-queries#optional-job-creation
From the code snippet for Go ^

	// The iterator provides information about the query execution.
	// Queries that were run in short query mode will not have the source job
	// populated.
	if it.SourceJob() == nil {
		fmt.Fprintf(w, "Query was run in optional job mode.  Query ID: %q\n", it.QueryID())
	} else {
		j := it.SourceJob()
		qualifiedJobID := fmt.Sprintf("%s:%s.%s", j.ProjectID(), j.Location(), j.ID())
		fmt.Fprintf(w, "Query was run with job state.  Job ID: %q, Query ID: %q\n",
			qualifiedJobID, it.QueryID())
	}

gcf-merge-on-green bot pushed a commit that referenced this pull request May 27, 2025
🤖 I have created a release *beep* *boop*
---


## [1.69.0](https://togithub.com/googleapis/google-cloud-go/compare/bigquery/v1.68.0...bigquery/v1.69.0) (2025-05-27)


### Features

* **bigquery/analyticshub:** Add support for Analytics Hub & Marketplace Integration ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b))
* **bigquery/analyticshub:** Adding allow_only_metadata_sharing to Listing resource ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b))
* **bigquery/analyticshub:** Adding CommercialInfo message to the Listing and Subscription resources ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b))
* **bigquery/analyticshub:** Adding delete_commercial and revoke_commercial to DeleteListingRequest and RevokeSubscriptionRequest ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b))
* **bigquery/analyticshub:** Adding DestinationDataset to the Subscription resource ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b))
* **bigquery/analyticshub:** Adding routine field to the SharedResource message ([2aaada3](https://togithub.com/googleapis/google-cloud-go/commit/2aaada3fb7a9d3eaacec3351019e225c4038646b))
* **bigquery:** Add support for dataset view and update modes ([#12290](https://togithub.com/googleapis/google-cloud-go/issues/12290)) ([7c1f961](https://togithub.com/googleapis/google-cloud-go/commit/7c1f9616b7ea95436582eb3c40c94e6bd9b48610))
* **bigquery:** Job creation mode GA ([#12225](https://togithub.com/googleapis/google-cloud-go/issues/12225)) ([1d8990d](https://togithub.com/googleapis/google-cloud-go/commit/1d8990dbf2563a5fbc96769ac9c6ea4ed06b239e))

---
This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants