Datasources API

About the Datasources API

CloudWisdom’s Datasources API can be used to get a list of datasources, create new ones, edit their settings, and remove them from CloudWisdom. You can test these endpoints by visiting our Swagger page and by clicking the interactive buttons below.

GET Use this endpoint to get a list of all datasources as well as their properties, elements, and collectors.

Parameters

Parameter

Parameter Type

Data Type

Description

includeElements

query

boolean

Includes or excludes elements in the response body.

Request URL

https://app.metricly.com/datasources?includeElements={boolean}

CURL

In the following CURL example, datasources are requested without elements (includeElements).

      curl -X GET --header 'Accept: application/json' 'https://app.metricly.com/datasources?includeElements=false'


    

Response Body

The following response body returns datasources without elements. This example has been shortened. Notice that datasources include collectors and list properties that define what is and is not ingested into CloudWisdom.

{
  "dataSources": [
    {
      "id": 25186,
      "name": "ExampleRuby",
      "type": "RUBY",
      "properties": {
        "filterTagTypeOld": "include",
        "awsAuthentication": "role"
      },
      "enabled": true,
      "deleted": false,
      "apiId": "cb75111111d71a9b9f9d9ed22b119c84b",
      "collectors": [
        {
          "id": 3888,
          "name": "RUBY",
          "packageEnabled": false,
          "lastSeen": null,
          "properties": {},
          "datasource": 25186,
          "package": null,
          "elements": []
        }
      ]
    },
    {
      "id": 33884,
      "name": "EXAMPLEAZURE",
      "type": "AZURE",
      "properties": {
        "asgFilterTagValue": "",
        "ebsFilterTagValue": "",
        "sqsFilteringEnabled": "false",
        "sqsEnabled": "true",
        "albFilterTagName": "",
        "ebsFilterTagName": "",
        "avmFilterTagType": "include",
        "rdsFilterTagValue": "",
        "ctmEnabled": "false",
        "azureSecretKey": "******",
        "ec2FilteringEnabled": "false",
        "lamFilterTagType": "include",
        "albFilterTagValue": "",
        "elbFilterTagType": "include",
        "dynFilterTagName": "",
        "sqsFilterTagType": "include",
        "avmFilteringEnabled": "false",
        "azureSubscriptionId": "cc8cad9f-ce22-1111-be6c-f444c9b2ab77",
        "s3FilterTagType": "include",
        "emrFilteringEnabled": "false",
        "lamFilterTagName": "",
        "sqsFilterTagName": "",
        "collectionGranularity": "5",
        "vmcEnabled": "true",
        "elbFilterTagName": "",
        "vmcFilterTagValue": "",
        "azureDomain": "111a4746-2ac3-0000-a6c8-bc9cdf67f222",
        "emrFilterTagName": "",
        "echFilterTagName": "",
        "knsFilteringEnabled": "false",
        "agwFilterTagName": "",
        "vmcFilteringEnabled": "false",
        "albFilterTagType": "include",
        "ebsEnabled": "true",
        "azureClientId": "0aea2a0c-2807-4077-a4d5-406fb79b3bff",
        "elbEnabled": "true",
        "echFilterTagValue": "",
        "asgEnabled": "true",
        "awsAuthentication": "role",
        "ec2FilterTagValue": "",
        "redEnabled": "false",
        "albEnabled": "false",
        "ecsFilterTagValue": "",
        "ecsFilterTagName": "",
        "emrEnabled": "false",
        "avmFilterTagValue": "",
        "s3FilterTagValue": "",
        "elbFilterTagValue": "",
        "avmEnabled": "true",
        "urls": "[{\"type\":\"URL\",\"url\":\"\",\"elementName\":\"\",\"generateElementName\":false,\"enableErrorTracking\":true}]",
        "s3Enabled": "false",
        "collectionPeriod": "5",
        "dynFilterTagType": "include",
        "ec2Enabled": "true"
      },
      "enabled": false,
      "deleted": false,
      "apiId": null,
      "collectors": [
        {
          "id": 5868,
          "name": "Virtual Machine Example",
          "packageEnabled": false,
          "lastSeen": null,
          "properties": {},
          "datasource": 33884,
          "package": null,
          "elements": []
        },
        {
          "id": 2927,
          "name": "Virtual Machine",
          "packageEnabled": false,
          "lastSeen": null,
          "properties": {},
          "datasource": 33884,
          "package": null,
          "elements": []
        }
      ]
    }
  ]
}

POST Use this endpoint to create a new datasource.

Tip

This method can be used to create one or many integrations and should only be used by an experienced power user as Virtana generally recommends creating new datasources in the UI. Properties associated to datasources are mapped as strings and cannot be validated, meaning typos are not caught and returned as errors.

Parameters

Parameter

Parameter Type

Data Type

Description

User-Agent

header

string

User-Agent

wrapper

body

JSON

JSON template of a datasource with updates included.

Request URL

https://app.metricly.com/datasources

CURL

In the following CURL example, a new datasource is created. Datasource requirements may vary per datasource type. For example, it is recommended you at least include the following properties to set up an AWS datasource:

  • awsAuthentication

  • awsAccountNumber

  • iamRole

  • bucketName

An existing IAM role and S3 bucket is required to submit these properties.

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'User-Agent: none' -d '{ \
   "dataSource": { \
     "name": "CloudWisdom AWS", \
     "type": "AWS", \
     "properties": { \
       "awsAuthentication": "role", \
       "awsAccountNumber": "600000000083", \
       "iamRole": "arn:aws:iam::600000000083:role/Netuitive", \
       "ec2Enabled": true, \
       "bucketName": "support-aws-cost" \
     } \
   } \
 }' 'https://app.metricly.com/datasources'

Swagger Payload

Use the following base template to create a minimal AWS datasource from swagger.

{
  "dataSource": {
    "name": "CloudWisdom AWS",
    "type": "AWS",
    "properties": {
      "awsAuthentication": "role",
      "awsAccountNumber": "600000000083",
      "iamRole": "arn:aws:iam::600000000083:role/Netuitive",
      "ec2Enabled": true,
      "bucketName": "support-aws-cost"
    }
  }
}

Response Body

The following response body returns a basic AWS datasource with minimal properties. You can add more via the UI or the PUT method.

{
  "dataSource": {
    "id": 25796,
    "name": "CloudWisdom AWS",
    "type": "AWS",
    "properties": {
      "awsAuthentication": "role",
      "awsAccountNumber": "600000000083",
      "iamRole": "arn:aws:iam::600000000083:role/Netuitive",
      "ec2Enabled": "true",
      "bucketName": "support-aws-cost",
      "regions": "us-east-1,us-east-2,us-west-1,us-west-2,ca-central-1,ap-south-1,ap-northeast-2,ap-southeast-1,ap-southeast-2,ap-northeast-1,eu-central-1,eu-west-1,eu-west-2,sa-east-1",
      "collectionGranularity": "5",
      "collectionOffset": "5",
      "collectionPeriod": "5",
      "threadPoolSize": "10"
    },
    "enabled": true,
    "deleted": false,
    "apiId": null,
    "collectors": []
  }
}

DELETE Use this endpoint to delete a datasource.

Parameters

Parameter

Parameter Type

Data Type

Description

User-Agent

header

string

User-Agent

id

path

long

id

Request URL

https://app.metricly.com/datasources/{datasourceId}

CURL

In the following CURL example, a datasource with the id12345 is deleted.

      curl -X DELETE --header 'Accept: application/json' --header 'User-Agent: none' 'https://app.metricly.com/datasources/12345'


    

Response Body

The following response body has no content. The success code is 204.

No Content

GET Use this endpoint to get information about one datasource.

Parameters

Parameter

Parameter Type

Data Type

Description

id

path

long

id

Request URL

https://app.metricly.com/datasources/{dataSourceId}

CURL

In the following CURL example, a datasource with the id 12345 is retrieved.

      curl -X GET --header 'Accept: application/json' 'https://app.metricly.com/datasources/12345'


    

Response Body

The following response body returns all properties and collectors associated to the datasource.

{
  "dataSource": {
    "id": 12345,
    "name": "NETUITIVE",
    "type": "AZURE",
    "properties": {
      "asgFilterTagValue": "",
      "mqFilteringEnabled": "false",
      "ebsFilterTagValue": "",
      "sqsFilteringEnabled": "false",
      "sqsEnabled": "true",
      "albFilterTagName": "",
      "efsCollectionFrequency": "5",
      "ebsFilterTagName": "",
      "avmFilterTagType": "include",
      "rdsFilterTagValue": "",
      "ctmEnabled": "false",
      "nlbFilteringEnabled": "false",
      "azureSecretKey": "******",
      "elbCollectionFrequency": "5",
      "ec2FilteringEnabled": "false",
      "lamFilterTagType": "include",
      "albFilterTagValue": "",
      "tagFilterTagName": "",
      "tagCollectionFrequency": "5",
      "elbFilterTagType": "include",
      "dynFilterTagName": "",
      "costExplorerEnabled": "true",
      "knsCollectionFrequency": "5",
      "sqsFilterTagType": "include",
      "avmFilteringEnabled": "false",
      "ctmCollectionFrequency": "5",
      "mqEnabled": "false",
      "azureSubscriptionId": "f9e8197a-1111-9999-3333-f571f5251411",
      "ctmFilterTagName": "",
      "s3FilteringEnabled": "false",
      "agwFilteringEnabled": "false",
      "tagFilterTagType": "include",
      "dynFilterTagType": "include",
      "r53FilterTagValue": "",
      "ec2Enabled": "true"
    },
    "enabled": false,
    "deleted": false,
    "apiId": null,
    "collectors": [
      {
        "id": 4321,
        "name": "Virtual Machine",
        "packageEnabled": false,
        "lastSeen": null,
        "properties": {},
        "datasource": 45243,
        "package": null,
        "elements": []
      },
      {
        "id": 7654,
        "name": "Virtual Machine (Classic)",
        "packageEnabled": false,
        "lastSeen": null,
        "properties": {},
        "datasource": 12345,
        "package": null,
        "elements": []
      }
    ]
}

PUT Use this endpoint to make updates to datasources.

Parameters

Parameter

Parameter Type

Data Type

Description

User-Agent

header

string

User-Agent

Id

path

long

Unique datasource ID.

wrapper

body

JSON

JSON template of a datasource with updates included.

Request URL

https://app.metricly.com/datasources/{id}

CURL

In the following CURL example, the datasource with id12345 is disabled by changing enabled to false.

curl -X PUT --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'User-Agent: none' -d '{ \
   "dataSource": { \
     "id": 12345, \
     "name": "ruby", \
     "type": "RUBY", \
     "properties": { \
       "filterTagTypeOld": "include", \
       "awsAuthentication": "role" \
     }, \
     "enabled": false, \
     "deleted": false, \
     "apiId": "cb7515557d71a9b9f9d9ed77b119c84b", \
     "collectors": [ \
     ] \
   } \
 }' 'https://app.metricly.com/datasources/25186'

You can use the following template to test this endpoint with Swagger. Select the method icon to open this specific endpoint.

Swagger Payload

{
  "dataSource": {
    "id": 12345,
    "name": "Ruby",
    "type": "RUBY",
    "properties": {
      "filterTagTypeOld": "include",
      "awsAuthentication": "role"
    },
    "enabled": false,
    "deleted": false,
    "apiId": "cb7515557d71a9b9f9d9ed77b119c84b",
    "collectors": [
    ]
  }
}

Response Body

The following response body returns more details about the datasource and allows the user to verity the values have been updated.

{
  "dataSource": {
    "id": 12345,
    "name": "Ruby",
    "type": "RUBY",
    "properties": {
      "filterTagTypeOld": "include",
      "awsAuthentication": "role"
    },
    "enabled": false,
    "deleted": false,
    "apiId": "cb7515557d71a9b9f9d9ed77b119c84b",
    "collectors": [
      {
        "id": 3888,
        "name": "RUBY",
        "packageEnabled": false,
        "lastSeen": null,
        "properties": {},
        "datasource": 25186,
        "package": null,
        "elements": []
      }
    ]
  }
}

Every datasource has properties that can be updated to enable or disable collection for different types of data. In this example we are going to disable collection for a particular AWS region.

  1. Build a CURL query to GET from /datasources. This allows us to see a full list of datasources and find the right id.

              curl -X GET --header 'Accept: application/json' 'https://app.metricly.com/datasources?includeElements=false'
    
            
  2. Select a datasource form the response body. Copy its id and build another GET query for /datasources/{id}

              curl -X GET --header 'Accept: application/json' 'https://app.metricly.com/datasources/12345'
    
            
  3. Review the response body. In particular, we are interested in the properties options. The properties in this example have been greatly reduced. Find the regions key.

    {
      "dataSource": {
        "id": 12345,
        "name": "AWS1-new",
        "type": "AWS",
        "properties": {
          "bucketName": "rcg-netuitive-support-aws-cost",
          "regions": "us-east-1,us-east-2,us-west-1,us-west-2,ca-central-1,ap-south-1,ap-northeast-2,ap-southeast-1,ap-southeast-2,ap-northeast-1,eu-central-1,eu-west-1,eu-west-2,sa-east-1",
          "lamFilterTagName": "",
          "sqsFilterTagName": "",
          "lamCollectionFrequency": "5",
          "collectionGranularity": "5",
          "efsFilteringEnabled": "false",
          "asgFilterTagType": "include",
          "rdsFilteringEnabled": "false",
          "rdsFilterTagName": "",
          "ec2CollectionFrequency": "5",
          "ebsFilterTagType": "exclude",
          "vmcEnabled": "true",
          "elbFilterTagName": "",
          "vmcFilterTagValue": "",
          "tagEnabled": "false",
          "nlbFilterTagName": "",
          "knsFilterTagValue": "",
          "ebsEnabled": "true",
          "iamRole": "arn:aws:iam::600000000003:role/Netuitive",
          "tagFilterTagValue": "",
          "elbEnabled": "false",
          "mqFilterTagValue": "",
          "echFilterTagValue": "",
          "asgEnabled": "true",
          "awsAuthentication": "role",
          "ec2FilterTagValue": "",
          "redEnabled": "false",
          "agwCollectionFrequency": "5",
          "r53FilterTagType": "include",
          "efsFilterTagType": "include",
          "knsFilterTagType": "exclude",
          "asgCollectionFrequency": "5",
          "redCollectionFrequency": "5",
          "dynFilteringEnabled": "false",
          "ecsFilterTagValue": "",
          "ecsFilterTagName": "",
          "emrCollectionFrequency": "5",
          "emrEnabled": "false",
          "avmFilterTagValue": "",
          "s3FilterTagValue": "",
          "ctmFilteringEnabled": "false",
          "elbFilterTagValue": "",
          "tagFilteringEnabled": "false",
          "avmEnabled": "true",
          "filterTagTypeOld": "exclude",
          "urls": "[{\"type\":\"URL\",\"url\":\"\",\"elementName\":\"\",\"generateElementName\":false,\"enableErrorTracking\":true}]",
          "nlbEnabled": "false",
          "r53FilterTagName": "",
          "avmCollectionFrequency": "5",
          "nlbCollectionFrequency": "5",
          "s3Enabled": "false",
          "collectionPeriod": "5",
          "awsAccountNumber": "600005000003"
        },
        "enabled": false,
        "deleted": false,
        "apiId": null,
        "collectors": [
        ]
      }
    }
  4. Build a CURL query for the PUT method for the endpoint /datasources/{id}. You’ll need to use the datasources’s id as a parameter. Remove the region us-east-1 from the list.

    curl -X PUT --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'User-Agent: none' -d '{ \
       "dataSource": { \
         "name": "AWS1-new", \
         "properties": { \
           "regions": "us-east-2,us-west-1,us-west-2,ca-central-1,ap-south-1,ap-northeast-2,ap-southeast-1,ap-southeast-2,ap-northeast-1,eu-central-1,eu-west-1,eu-west-2,sa-east-1} \
     } \
     }' 'https://app.metricly.com/datasources/12345
  5. The response body returns the whole datasource definition. Verify that the regions have updated.