Qubit Docs العربية

Amazon S3

Connect an Amazon S3 bucket, or any S3-compatible storage, so Qubit can load CSV or Parquet files from a folder on a schedule. All files in the folder load into one table.

How files become tables

  • Every CSV or Parquet file under the folder you enter loads into a single table named after the folder.
  • The first file defines the table's columns, so keep the same columns in every file in the folder.
  • To load several folders as separate tables, use the advanced Multiple tables (advanced, JSON) field described below.

Before you start

  • Create an IAM user for Qubit with read-only access to the bucket, for example permission to list the bucket and read its objects.
  • Create an access key for that user and keep the secret access key. AWS shows it only once.

Connect Amazon S3

  1. Open Data Catalog from the sidebar, click Add Data, then choose Connect a database.
  2. On the New Data Source page, click Amazon S3. It is under Cloud Storage.
  3. Enter a Connection Name.
  4. Fill in the fields:
    • AWS Access Key ID and AWS Secret Access Key: the access key you created.
    • Bucket Name: the bucket name only.
    • Folder (prefix): required. The folder path inside the bucket, for example exports/orders/.
    • File name pattern: optional. A regular expression matched against the full object key, for example \.csv$.
    • File type: Detect from extension (default), CSV, or Parquet.
    • AWS Region: the bucket's region. Defaults to us-west-2.
    • Custom endpoint URL: only for S3-compatible storage such as MinIO or Wasabi. Leave it empty for AWS.
    • Multiple tables (advanced, JSON): optional. A JSON list that loads several folders as separate tables. It overrides the folder above. For example: [{"table_name": "orders", "prefix": "exports/orders/", "search_pattern": "\\.csv$"}].
  5. Choose a Sync Schedule. The default is Daily.
  6. Click Create.

After you connect

Qubit checks that the required fields are filled in, saves the connection, and starts the first sync. It confirms that the key can read the bucket during that sync.

To check the connection at any time, open its menu in the Data Catalog and choose Test connection. If Last sync shows Failed, hover over it to see the reason, correct the details with Edit, then choose Sync now.