Documentation
Toggle Dark/Light/Auto modeToggle Dark/Light/Auto modeToggle Dark/Light/Auto modeBack to homepage
  1. Docs
  2. /
  3. Reference
  4. /
  5. Private Broker Setup
  6. /
  7. Frequently Asked Questions
Add-on

Frequently Asked Questions

What is the Sundeck Private Broker?

The private broker is a lightweight, stateless service that acts as a proxy between SQL clients and Snowflake, deployable entirely within a customer’s VPC. The Private Broker is provided as a Docker image, and thus is suitable to be hosted and managed in a variety of ways, from standalone to an orchestration framework.

How does scale-out and HA work?

We current support vertical scaling for the private broker. A single private broker instance can handle a large number of concurrent user sessions and queries (up to 1000s of concurrent sessions), and as such we recommend starting with a single standalone instance, or a single instance in a container orchestration framework (which can provide high-availability through automatic container respawns in the event a container goes down).

Customers are encouraged to have a conversation with Sundeck before implementing horizontal scaling (multiple instances of Private Broker running concurrently behind a single load balancer), as there are a couple of potential known issues:

  1. The Private Broker caches session state in memory. This means that session affinity / session pinning must be implemented such that requests from the same client session terminate on the same Private Broker instance to avoid stale session state.
  2. Forced pinned-table refreshes which are triggered via DDL (via ALTER SUNDECK TABLE ... REFRESH ... commands) will only affect the Private Broker instance these requests land on. Forced refreshes of pinned tables are not supported currently in Private Broker. The default timer-based refreshes are effective, however are not synchronized across all Private Broker instances.

What happens if the Private Broker container crashes or restarts?

There is no impact for existing user sessions. The PB rebuilds all the required session state on its restart.

Can I switch Sundeck tenants within an instance of Private Broker?

Currently, the Sundeck Private Broker supports only a single Sundeck SaaS tenant at a time.

Does the private broker have any storage or state ? Does it have any external dependencies?

The Private Broker is stateless and does not persist any state. It has no dependencies except the obvious ones (Snowflake, Sundeck’s SaaS Control Plane, and the container environment hosting the Private Broker container).

How do Pinned Tables work in Private Broker?

The private broker fetches the metadata related to Pinned Tables (eg. table name, column list) from the Sundeck SaaS Control Plane, then fetches the contents of the Pinned Table(s) from Snowflake periodically (based on the refresh interval) using Sundeck Service Account credentials which are passed to the Private Broker on startup (via environment variables).

User will not perceive any difference in the behaviour of Pinned Tables between Sundeck’s SaaS Broker solution and Private Broker.

What happens to Pinned Table data if the Private Broker restarts?

At Private Broker startup, data is fetched from Snowflake for all Pinned Tables and pinned into broker memory. All Pinned Table data is loaded before the Broker is up and able to accept incoming queries (guaranteeing that no queries arrive at the Broker before it has the data needed to make decisions).

While the broker is populating the Pinned Table(s) during startup, all incoming user requests to Private Broker will fail/timeout with a connection error like “Could not connect to snowflake backend after 2 attempt(s)”. Normally, the Pinned Table refresh is quite fast (a few seconds) so this should not impact user queries.

Some Pinned Tables may not be populated during startup if:

  1. An error is encountered while populating the pinned table from Snowflake
  2. If populating the pinned table takes longer than 5 mins