How to Save and Load Algorithm State Using the ObjectStore in QuantConnect

To save data from your algorithm for later use (e.g., to persist state between backtests, pass data to live trading, or store machine learning models), you should use the ObjectStore feature in QuantConnect.

The ObjectStore is a key-value storage system that allows you to save strings, JSON, bytes, or XML files. These objects persist in your project environment and can be retrieved in subsequent runs.

Implementation Using ObjectStore

The following example demonstrates how to:

1. Check if data already exists using Contains.
2. Load and deserialize data using Read.
3. Save and serialize data using Save.

# region imports
from AlgorithmImports import *
import json
# endregion

class DataPersistenceAlgorithm(QCAlgorithm):

    def initialize(self):
        self.set_start_date(2023, 1, 1)
        self.set_end_date(2023, 6, 1)
        self.set_cash(100000)
        
        self.add_equity("SPY", Resolution.DAILY)
        
        # Define a unique key for your data
        self.storage_key = "strategy_state_data"
        
        # Initialize a container for our data
        self.strategy_data = {
            "total_trades": 0,
            "cumulative_profit": 0.0,
            "last_processed_time": str(self.time)
        }

        # 1. Check if the key exists in the ObjectStore
        if self.object_store.contains_key(self.storage_key):
            # 2. Read the data (returns a string/bytes)
            stored_content = self.object_store.read(self.storage_key)
            
            # Deserialize JSON string back to a dictionary
            try:
                self.strategy_data = json.loads(stored_content)
                self.debug(f"Data loaded successfully: {self.strategy_data}")
            except Exception as e:
                self.error(f"Failed to parse stored data: {e}")
        else:
            self.debug("No existing data found. Starting fresh.")

    def on_data(self, data: Slice):
        if not self.portfolio.invested:
            self.set_holdings("SPY", 0.5)
            self.strategy_data["total_trades"] += 1

    def on_end_of_algorithm(self):
        # Update the state before saving
        self.strategy_data["cumulative_profit"] = self.portfolio.total_profit
        self.strategy_data["last_processed_time"] = str(self.time)

        # 3. Serialize the dictionary to a JSON string
        json_string = json.dumps(self.strategy_data, indent=4)
        
        # Save to ObjectStore
        self.object_store.save(self.storage_key, json_string)
        
        self.debug(f"Data saved to ObjectStore under key '{self.storage_key}'")
        self.debug(f"Final Content: {json_string}")

Key Methods Explained

• self.object_store.contains_key(key): Returns a boolean indicating if the specified key exists in the storage.
• self.object_store.read(key): Retrieves the content associated with the key. It returns a string (or bytes if specified).
• self.object_store.save(key, content): Saves the content to the storage. If the key exists, it overwrites the previous data.
• self.object_store.get_file_path(key): Returns the local file path of the object, which is useful if you need to load the file using a library like joblib or pickle for Machine Learning models.

Common Use Cases

1. Machine Learning: Training a model in a backtest, saving the model file, and loading it in Live Trading to avoid retraining.
2. State Persistence: Remembering the high-water mark or specific trade counters when restarting an algorithm.
3. Parameter Optimization: Storing results from one research notebook to be used in a backtest.

Q&A: Data Persistence in QuantConnect

Q: Can I use Python's built-in open() and write() functions?
A: In the cloud environment, local file storage is ephemeral. While you can write to the local directory during execution, these files are deleted once the algorithm stops. To persist data permanently across different runs, you must use the ObjectStore.

Q: Is there a limit to how much data I can store?
A: Yes, the ObjectStore has storage limits based on your organization's tier. Large datasets should be managed carefully, or you should consider cleaning up old keys using self.object_store.delete(key).

Q: Can I access ObjectStore data from the Research Environment?
A: Yes. The ObjectStore is accessible in the Research Environment (Jupyter Notebooks) using the qb.object_store object (where qb is the QuantBook instance). This allows you to analyze data saved by your algorithm or prepare data for your algorithm to read.

Q: How do I save binary data, like a trained Scikit-Learn model?
A: You can save binary data by converting it to bytes. For libraries like joblib or pickle, it is often easier to save the file locally first, then read the bytes and save to ObjectStore:

# Example for saving a model
import joblib
# Save locally
joblib.dump(model, "model.joblib")
# Read bytes and save to ObjectStore
byte_data = self.object_store.read_bytes("model.joblib") # Reads local file if not in store
self.object_store.save_bytes("my_model_key", byte_data)