1 Service Discovery

So far for every target, we have been manually specifying the IP address and port in their scrape configuration. This approach doesn't scale for larger systems.

Prometheus solves it by using service discovery: automated mechanisms to detect, classify, and identify new and changed targets.

Variety of ways to use service discovery:

  • Receiving list of targets from files populated via configuration management tools.
  • Quering an API like AWS API.
  • Using DNS to return list of targets.

1.1 File-base Discovery

Edit prometheus.yml 

  scrape_configs:
   - job_name: 'node'
     file_sd_configs:
      -files:
       -targets/nodes/*.json
       refreh_interval: 5m

   $ cat targets/nodes/nodes.json
   [{
   "targets": ["138.197.26.39:9100","138.197.26.39:9101"]
   }]

We could also add labels: 

   $ cat targets/nodes/nodes.json
   [{
   "targets": ["138.197.26.39:9100","138.197.26.39:9101"],
   "labels": { "datacenter": "nj"}
   }]

1.2 Inbuilt service discovery plugins

Some platforms are supported by native service discovery integrations. These ship with Prometheus:

1.3 DNS service discovery

This allows you to specify a list of DNS entries and then query records in those entries to discover a list of targets. It relies on querying A, AAAA or SRV records.