Step-by-step deployment guide
We highly recommend using our SQL gateway instead of this import agent.
1. Configuring the agent
To configure your import agent, mount a JSON configuration file at /app/config/config.hjson with the properties listed below. Refer to the following attachments for a starting point:
These example configurations will simply import some dummy data from different databases. Edit the following properties in your file to configure the import agent to your needs:
cronPattern
string
Optional CRON pattern describing when the import agent should run (refer to https://crontab.guru for example patterns); omit if you just want to run the agent once
entityTypes
list[object]
Describes how to import entities of a specific type into Elimity Insights
entityTypes[].attributes
list[object]
Describes how to import attributes of this entity type's entities
entityTypes[].attributes[].description*
string
Description of the attribute type for which the agent should import assignments
entityTypes[].attributes[].id
string
Unique identifier of the attribute type for which the agent should import assignments
entityTypes[].attributes[].name*
string
Human-readable name of the attribute type
entityTypes[].attributes[].type
string
Data type of the attribute type, one of "boolean", "date", "dateTime", "number", "string" or "time"
entityTypes[].icon*
string
Icon of the entity type
entityTypes[].id
string
Unique identifier of the entity type for which the agent should import entities
entityTypes[].plural*
string
Plural form of the entity type's name
entityTypes[].query
string
Query that the import agent should send to the configured SQL instances, each resulting row corresponds to an entity of this type; refer to our documentation about the query format for additional information
entityTypes[].singular*
string
Singular form of the entity type's name
insightsSourceId
number
Elimity Insights source identifier; we'll create one in step 2
insightsSourceToken
string
Elimity Insights source token; we'll generate one in step 2
insightsUrl
string
URL of your Elimity Insights server
db2Dsns**
list[string]
List of connection strings that the import agent should use to connect with your IBM Db2 instances; we recommend connecting with a dedicated read-only user
mssqlDns**
list[string]
List of connection strings that the import agent should use to connect with your Microsoft SQL Server instances (we use Microsoft's official Go MSSQL driver behind the scenes, refer to its documentation on supported connection string formats); we recommend connecting with a dedicated read-only user (e.g. by only assigning the db_datareader role)
mySqlDsns**
list[string]
List of connection strings that the import agent should use to connect with your MySQL instances (we use the de facto standard MySQL driver for Go behind the scenes, refer to its documentation on supported connection string formats); we recommend connecting with a dedicated read-only user
oracleDsns**
list[string]
List of connection strings that the import agent should use to connect with your Oracle Database instances (we use the de facto standard Oracle Database driver for Go behind the scenes, refer to the implementation of their connection string parser); we recommend connecting with a dedicated read-only user
relationshipTypes
list[object]
Describes how to import relationships between entities of two specific types into Elimity Insights
relationshipTypes[].attributes
list[object]
Describes how to import attributes of this relationship type's relationships
relationshipTypes[].attributes[].description*
string
Description of the attribute type for which the agent should import assignments
relationshipTypes[].attributes[].id
string
Unique identifier of the attribute type for which the agent should import assignments
relationshipTypes[].attributes[].name*
string
Human-readable name of the attribute type
relationshipTypes[].attributes[].type
string
Data type of the attribute type, one of "boolean", "date", "dateTime", "number", "string" or "time"
relationshipTypes[].from
string
Unique identifier of the entity type from which the relationships start
relationshipTypes[].query
string
Query that the import agent should send to the configured SQL instances, each resulting row corresponds to a relationship between two entities of the configured types; refer to our documentation about the query format for additional information
relationshipTypes[].to
string
Unique identifier of the entity type where the relationships end
transactionIsolationLevel
number
Represents which isolation level the agent should use for the transaction in which it combines all queries; refer to our documentation about transaction isolation levels for additional information
*These properties do not affect the importing functionality, they are only necessary to generate a data model export file when running the import agent with the -print-data-model-export flag.
**The import agent processes each connection string independently and simply concatenates the results to form a single domain graph.
2. Creating a source in Elimity Insights
The import agent also includes some functionality to help you with creating a source in Elimity Insights. Assuming you correctly configured the agent as explained in the previous step, the following command will write a data model export file to data-model-export.json:
You can now upload this file to Elimity Insights when creating a new custom source. Copy the resulting source's identifier and token to the insightsSourceId and insightsSourceToken properties in the agent's configuration file.
3. Deploying the agent
Having configured the agent and having created a source in Elimity Insights, you can now deploy the agent to regularly import data from your SQL databases and upload it to Elimity Insights. Since we distribute the agent as a Docker image, our recommendation for deployment is to use a CaaS solution like Google Cloud Run or Azure Container Apps. If that's not an option, you can also manually deploy the image on e.g. Windows Server. Refer to our documentation about gateways and import agents for additional details.
4. Following up on the import
The import agent outputs logs to indicate its progress, for a manual Windows Server deployment you can check these with docker-compose logs.
Last updated