• Github link

    $ node_exporter --version
    node_exporter, version 1.0.1 (branch: HEAD, revision: 3715be6ae899f2a9b9dbfd9c39f3e09a7bd4559f)
    build user:       root@1f76dbbcfa55
    build date:       20200616-12:44:12
    go version:       go1.14.4
  

• The node_exporter exporter runs, by default on 9100 and exposes metrics on the /metrics path.

• The textfile collector is very useful because it allows us to expose custom metrics.
• The collector works by scanning files in a specified directory, extracting any string that are formatted as Prometheus metrics and exposing them to be scraped.

    $ mkdir textfile_collector
    $ # emacs and add metric file
    $ bat textfile_collector/metadata.prom
  ───────┬───────────────────────────────────────────────────
         │ File: textfile_collector/metadata.prom
  ───────┼───────────────────────────────────────────────────
     1   │ metadata{role="docker_server",datacenter="NJ"} 1
  

  Metric name: metadata

    $ node_exporter --collector.textfile.directory ./textfile_collector --collector.systemd --collector.systemd.unit-whitelist="(docker|ssh|rsyslog).service"
  

  You can see the raw metrics at http://localhost:9100/metrics

  Now can you edit the prometheus.yml configuration file to scrape the above metrics:

    scrape_configs:
     - job_name: 'prometheus'
       static_configs:
       -targets: ['localhost:9090']
     - job_name: 'node'
       static_configs:
       -targets: ['localhost:9100']
  

Edit the prometheus.yml configuration file:

   - job_name: 'node'
     static_configs:
      -targets: ['localhost:9100']
     params:
      collect[]:
       -cpu
       -meminfo

• Changing a label or adding a new label creates a new time series. So, labels should be used judiciously and remain as constant as possible.

We already get two topological labels for free with every metric: job and instance.

Vector matches attempt to find a matching element in the right hand side vector for each entry on the left hand side.

Types:

• One to one matches: Finds a unique pair of entries from each side of the operation.
• Many to one matches: are where each vector element on the "one" side can match with multiple elements on the "many" side.

Recording rules are a way to compute new time series, particularly aggregated time series, from incoming time series. Use case:

• Produce aggregates across multiple time series.
• Precompute expensive queries.
• Produce a time series that we could use to generate an alert.

   $ mkdir -p rules
   $ cd rules
   $ touch node_rules.yml

Edit prometheus.yml:

   rule_files:
   - "rules/node_rules.yml"

   $ cat rules/node_rules.yml
   groups:
   - name: node_rules
     rules:
     - record: instance:node_cpu:avg_rate5m
       expr: 100 -avg (irate(node_cpu_seconds_total{job="node",mode="idle"}[5m])) by (instance) * 100

$ ./grafana-server

• Visit http://localhost:3000/login
• Default Credentials: admin for both username and passwords.
• Add Prometheus data source.
• You can import dashboards from the data source page of prometheus.
• You can view those dashboards by going to the dashboards page!