bigquery-bigframes

BigFrames (BigQuery DataFrame) basics

BigFrames is a Python library that lets you take advantage of BigQuery
data processing by using familiar Python APIs.

Dataframe API best practices

• Stay in the Cloud: Perform data cleaning, transformation, and analysis
  via BigFrames methods to leverage BigQuery's scale rather than downloading
  data.
• Prefer partial ordering mode: Enable partial ordering mode right after
  importing BigFrames. This speeds up data processing significantly by relaxing
  row-sequence constraints.

  import bigframes.pandas as bpd
  bpd.options.bigquery.ordering_mode = 'partial'
  

• Use peek() for data preview: Use peek(n) to preview data instead of
  head(n). peek(n) randomly samples n rows and is significantly faster.
  head(n) returns rows in strict order and fails in partial ordering mode
  unless the DataFrame has been explicitly sorted.
• Avoid materializing data locally: Methods like to_pandas() download all
  data to client memory, bypassing BigQuery’s distributed computation and
  risking Out of Memory (OOM) errors. Do not materialize data locally unless:
  • The dataset is small enough to fit safely in memory.
  • An error message explicitly requires local materialization.
• Prefer Dataframe API over SQL queries: Do not write raw SQL queries via
  read_gbq() if a DataFrame/Series method achieves the same result, as it
  breaks the Pandas abstraction and prevents lazy query execution.
• Accessors over UDFs/Lambdas:
  • Use built-in accessors (e.g., df.col.str.*, df.col.dt.*) instead of
    remote User Defined Functions (UDFs). UDFs require extra resources and
    time to deploy.
  • Do not use lambdas with Series.map() or DataFrame.apply(). These
    methods do not accept functions without udf or remote_function
    decorators.

  # Avoid:
  df["upper"] = df["name"].map(lambda x: x.upper())
  
  # Prefer:
  df["upper"] = df["name"].str.upper()
  

• Schema Verification: Do not assume the schema of intermediate outputs.
  Proactively verify schemas using .dtypes and inspect sample records using
  display() with .peek().
• Visualization: Plot directly from the BigFrames DataFrame/Series when
  possible. BigFrames is compatible with Matplotlib and Seaborn. If direct
  plotting fails, use the .plot accessor. If the dataset is too large to plot,
  aggregate or sample the data before calling
  .to_pandas() to plot locally.

Machine Learning

• Use bigframes.bigquery.ml package: Do not use Scikit-learn or other ML
  libraries with BigQuery DataFrames. Standard Scikit-learn models require
  bringing data into local client memory, whereas bigframes.bigquery.ml
  delegates training directly to BigQuery's scalable ML engine. Import functions
  from bigframes.bigquery.ml.

Reference Directory

• Linear Regression: Train a linear
  regression model to predict numerical values.
• Logistic Regression: Train a logistic
  regression model to predict boolean values.

BigFrames ML (Legacy)

The BigFrames ML package (bigframes.ml) is a legacy package that mimics the
scikit-learn API but is no longer recommended for new projects. Only use this
package if the user explicitly requests BigFrames ML.

• Legacy Imports: When legacy BigFrames ML is requested, import tools and
  classes from bigframes.ml instead of bigframes.bigquery.ml.
• DataFrame Return on Prediction: Unlike Scikit-learn, BigFrames'
  predict() method always returns a DataFrame containing both predictions
  and features, rather than a single series of predictions.
• No random_state: Do not pass a random_state argument when
  instantiating BigFrames ML models, as this parameter is not supported in the
  BigFrames ML package.
• Automatic Scaling: Do not use OneHotEncoder or StandardScaler unless
  explicitly requested, as scaling is handled automatically.
• Hyperparameter Tuning: Write custom loops for hyperparameter tuning, as
  BigFrames lacks GridSearchCV or RandomizedSearchCV.
• ARIMA Plus (Forecasting):
  • Import from bigframes.ml.forecasting.
  • Sort data chronologically and split around a timepoint before training.
  • Ensure the prediction horizon is less than or equal to the training
    horizon.
• PCA: BigFrames' PCA class lacks a transform() method. Use predict()
  instead.
• Model Persistence: To persist a model, use model.to_gbq(). To load a
  persisted model, use bpd.read_gbq_model().