Big Objects, Data Retention & Archive Architecture
A Salesforce Big Object is not an archive strategy
An archive is defined by the questions it must answer
A team moves ten years of service history into a custom Big Object and celebrates the reduction in standard storage. Months later, Legal asks for every interaction involving one customer during a disputed period. The chosen index starts with region and product, the user interface cannot search by customer alone and the export process has never been rehearsed. The records still exist, but the organisation cannot retrieve defensible evidence when it matters.
The JSBC Labs position is that a Salesforce Big Object is a storage technology—not an archive strategy. Salesforce provides a distributed, non-transactional data store for very large volumes with predictable index-based access. A complete archive also needs an access contract, ownership, security model, migration evidence, retention rules and an executable deletion process. If those decisions arrive after loading, the archive can become an expensive data graveyard.
Classify the lifecycle before choosing the platform
Start by separating active, warm, cold and expired data. Salesforce Well-Architected guidance places frequently used operational data in standard objects, occasionally accessed data in warm patterns, and predictable high-volume retention in Big Objects or external archives. Expired data should be deleted rather than retained indefinitely. That classification is more useful than a blanket rule such as move everything older than two years.
For each dataset, document the retention authority, required retrieval time, expected query patterns, permitted users, legal-hold behaviour and final disposition. Big Objects suit records that must remain on-platform at very high volume and can be accessed through known keys. External object storage or a data platform may be better when the primary needs are low-cost retention, broad analytics, immutable evidence or search patterns that change frequently. The cheapest storage choice is not necessarily the lowest-cost operating model.
Design retrieval before committing the index
A custom Big Object index is its composite primary key and its principal access path. Salesforce requires SOQL filters to use indexed fields in index order without gaps. Earlier fields use equality, while the last supplied field supports a limited range of operators. After deployment, the index cannot be edited or deleted; changing it means starting with a new Big Object. Index design is therefore a long-lived product decision, not a late performance tweak.
Collect real retrieval scenarios from service, finance, audit, legal and engineering. Convert each into a query shape, then test whether one ordered index supports the required leading fields and ranges. Be explicit about cardinality, time boundaries and tenant or business-unit separation. If unrelated query families require different leading keys, consider purpose-specific Big Objects or an external search layer instead of forcing one index to serve contradictory needs. Prototype with production-like volumes before committing the metadata.
Remove live behaviour deliberately
Big Objects do not behave like ordinary Salesforce objects. Salesforce documents that they do not support standard sharing rules, triggers, flows, processes or the Salesforce mobile app, and they use custom Lightning or Visualforce components rather than standard object pages, list views and home pages. Moving a record can therefore remove automation, collaboration, search and user-interface behaviour that teams assumed belonged to the data itself.
Create a dependency map before migration: automation, validation, lookups, reports, integrations, attachments, permissions and user journeys. Decide which capabilities end at archival, which must be reproduced and which need a summary retained on an active record. A common pattern is to keep an operational synopsis in the CRM, retrieve detailed history on demand and prohibit archived records from driving live workflow. Make that boundary visible to users rather than presenting a partial archive as if it were the original record.
Treat loading as a reconciled migration
Salesforce supports Bulk API and Bulk API 2.0 for loading and querying Big Objects. Inserts are idempotent by index, which helps safe retries, but the platform also warns that distributed writes can produce partial batch failures. Apex uses Database.insertImmediate(), returns SaveResult errors and treats Big Object DML as callouts. These mechanics reward deliberate batch identity and reconciliation rather than fire-and-forget export scripts.
Define source selection, transformation, index construction and an immutable migration-run identifier. Reconcile source counts, accepted rows, rejected rows, duplicates and sampled field values before deleting anything from the source object. Preserve failure files and rerun evidence. Test late-arriving updates and overlapping batches so an idempotent write does not silently replace the wrong representation. Archive migration should have the same cutover discipline as a production data release, including a pause point and a recovery path.
Rebuild security for the archived state
Big Objects support object and field permissions, but not the regular sharing model used by standard and custom objects. Salesforce also states that Shield Platform Encryption is not supported for custom Big Object fields; data archived from an encrypted standard or custom field is stored as clear text in the custom Big Object, although the Big Object store has encryption at rest. That distinction can materially change the control posture of sensitive data.
Reclassify the archived dataset and prove least privilege for every retrieval path, integration and custom component. Avoid copying fields merely because they exist in the source. Tokenise or omit sensitive values where the business question permits it, and test access with real personas. Record which controls changed during migration and obtain security approval before source deletion. Retention for compliance does not justify widening visibility or preserving personal data beyond its lawful purpose.
Design reporting as a separate data product
Salesforce does not present Big Object data directly in ordinary reports and dashboards. Its implementation guidance recommends using Bulk API to extract a smaller representative working dataset into a custom object when platform reporting is required. That means a successful archive can still fail stakeholders if they expected ad hoc dashboards over billions of historical rows.
Define which questions need detailed retrieval, which need aggregates and which need analytical exploration. Materialise governed summaries for common operational measures and send broad analytical workloads to a platform designed for them. Give every derived dataset an owner, freshness target and reconciliation test against the archive. Do not rebuild the original storage problem by copying an ever-growing working set back into standard objects simply to satisfy an undefined reporting request.
Make retention and deletion executable
An archive without deletion is uncontrolled accumulation. Salesforce supports deleting custom Big Object data through Database.deleteImmediate() or SOAP deleteByExample(), but deletion must identify every index field; partial indexes and wildcards are not supported. That makes final disposal another consequence of index design. A retention rule that cannot be translated into the index may be operationally difficult to enforce at scale.
Maintain a retention catalogue that connects each record class to its authority, trigger date, legal holds, deletion method and evidence owner. Rehearse retrieval, correction and deletion with production-scale samples. Monitor load failures, query latency, reconciliation variance, capacity and overdue disposal. A trustworthy archive proves three things continuously: the right records entered, authorised people can retrieve them within the promised time and records leave when their purpose expires.
Official references
- Salesforce Developers — Big Objects
- Salesforce Developers — Define and Deploy Custom Big Objects
- Salesforce Developers — SOQL Object Limits and Limitations
- Salesforce Developers — Big Objects Best Practices
- Salesforce Developers — Populate a Custom Big Object
- Salesforce Developers — Delete Data in a Custom Big Object
- Salesforce Developers — View Big Object Data in Reports and Dashboards
- Salesforce Architects — Storage and API Cost Optimization Patterns