Databricks

Overview

Granulate gAgent installation on Databricks allows seamless integration with plug-and-play support. The gAgent Databricks installation enables deploying Granulate agent on Databricks cluster nodes by incorporating Granulate's bash CLI installation to the Databricks cluster-scoped init scripts.

Installation

Granulate gAgent can be added to a Databricks cluster-scoped init scripts directly from the cluster's DBFS or from an S3 bucket.

Add the init script as a DBFS directory-

  1. 1.
    Create a DBFS directory you want to store the init script in by running:
1
dbutils.fs.mkdirs("dbfs:/databricks/scripts/")
Copied!
2. Create Granulate agent installation script in the directory. Make sure to fill in your customer-specific Download Bucket and Client ID.
  • For all-purpose existing clusters, use the following installation script (service name is extracted based on the cluster name):
1
dbutils.fs.put("/databricks/scripts/granulate_run_agents.sh","""
2
#!/bin/bash
3
set -e
4
export CLIENT_ID="aLU+0EiAhKU8uP7nxR86Zg=="
5
curl -s https://s3.amazonaws.com/<Download Bucket>/granulate_run_gagent.sh | sudo \
6
CLIENT="${CLIENT_ID}" \
7
SERVICE="databricks-$(echo ${DB_CLUSTER_NAME} | tr ' ' '-' | tr '.' '-')" \
8
bash
9
echo "Installation Successful"
10
exit 0
11
""", True)
Copied!
  • For job clusters, create an installation script for each job, and make sure to set the Job Name according to the name of the job:
1
dbutils.fs.put("/databricks/scripts/granulate_run_agents_<Job Name>.sh","""
2
#!/bin/bash
3
set -e
4
export CLIENT_ID="aLU+0EiAhKU8uP7nxR86Zg=="
5
curl -s https://s3.amazonaws.com/<Download Bucket>/granulate_run_gagent.sh | sudo \
6
CLIENT="${CLIENT_ID}" \
7
SERVICE="databricks-<Job Name> | tr ' ' '-' | tr '.' '-')" \
8
bash
9
echo "Installation Successful"
10
exit 0
11
""", True)
Copied!

Configure a cluster-scoped init script using the UI

To use the cluster configuration page to configure a cluster to run an init script:
  1. 1.
    On the cluster configuration page, click the Advanced Options toggle.
  2. 2.
    At the bottom of the page, click the Init Scripts tab.
  3. 3.
    In the Destination drop-down, select a destination type. In the example in the preceding section, the destination is DBFS.
  4. 4.
    a. Enter the following path for general-purpose clusters:
    dbfs:/databricks/scripts/granulate_run_agents.sh
    b. Enter the following path for job clusters (make sure to set the Job Name):
    dbfs:/databricks/scripts/granulate_run_agents_<Job Name>.sh
Last modified 22d ago
Copy link