You've successfully subscribed to Smartcodehub ™ Blog
Great! Next, complete checkout for full access to Smartcodehub ™ Blog
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Sensefull Logging | Nest js Recipies

Sensefull Logging | Nest js Recipies

Neeraj Dana
Neeraj Dana

An Application without logs is very dangerous. and it becomes more dangerous when we do not monitor the logs. This post will be a mixture of my experience of facing such issues and one way to resolve from thousands of way available .

let's start with the problem ...

Some Problems

Recently there was one Project which is a mobile app talking to different other mobile apps (all are from the same development team ) and there was two APIs which were monolithic so the problem was we were encountering a lot of issues in the apps and we were actually not able to find the root cause unless the developer debugs it. so there was a tight dependency on app developers even to trace the problem weather it is servers,  APIs, or app.

That was the first time we realize we should have a logging system which everyone has access to and anyone (the internal team ) can actually just see at some platform and find out what might went wrong . and notify the concerned person to resolve

There was one more case we were actually not notified or trace whether our hosted servers are up or not unless we use the app or make a request calls. that can also make a very bad impression on the client when you have a public-facing app

Let's say there are 100 issues in the app which the end-user is facing out of that 100 users only 30 -40 % of users will take time to write to your support team about their experience. out of rest 60% of users, 30% of users may uninstall the app and may give a negative recommendation to others. from a business perspective that is a very big loss.

Solution

As you might have already guessed from the title almost all this issues can be caught with simple logging system. ill just saw what I think was best for that use cases your feedback is most welcomed


So i used Grafana Loki
Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Grafana is used to monitor everything from Prometheus & Graphite metrics, logs, applications, power plants, beehives, sourdough starters and custom data sources.

Will also be using Nats Streaming server

Setup Grafana And Loki

I will be using simple docker system

Step 1: Create Network
we require a docker network on which our different applications can work

docker network create smartcodehubnetwork

Step 2: Setup Nats Server

docker run -d --network smartcodehubnetwork -p 4222:4222 -p 8222:8222 -p 6222:6222 --name nats-server -ti nats:latest

Step 3: Setup Loki Grafana and promtail Instances

docker container run -d --name lokiinstance --network smartcodehubnetwork -p 3100:3100 grafana/loki:latest

docker container run -d --name promtailinstance --network smartcodehubnetwork -v /var/log:/var/log grafana/promtail:latest

docker container run -d --name grafanainstance --network smartcodehubnetwork -p 3060:3000 grafana/grafana:latest

now you can actually go to localhost:3060 to see the default grafana screen use admin as username and password . Change the password and rockon your grafana is up

Step 4 : Connect Grafana with loki

Go to setting icon (configuration on the dashboard ) then go to data sourcesClick on the add data source and select loki and add the name as per the application and  in URL add http://lokiinstance:3100

Note: as we can see we can access the services by name when they are on same network

click on save&test if it shows green once done you are all set up to see logsfrom different part of the system

Step 5: Create a microservice for logging  

ill be using nest js as i like it its in typescript well structured and developer friendly but you can use any thing

You can read about creating nest micoservice here .

main.ts

import { NestFactory } from '@nestjs/core';
import { Transport, MicroserviceOptions } from '@nestjs/microservices';
import { AppModule } from './app.module';
import { hostname } from 'os';

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(
    AppModule,
    {
      transport: Transport.NATS,
      options: {
        url: `nats://nats-server:4222`,
      },
    },
  );
  app.listen(() => console.log('Microservice is listening'));
}
bootstrap();

as you can see on line url: nats://nats-server:4222 we are again using nats by name as our logging service will also run on the same network

appcomponent.ts

import { Controller, Get } from '@nestjs/common';
import { GrafanaLoggerService } from './app.service';
import { EventPattern } from '@nestjs/microservices';
@Controller()
export class AppController {
  constructor(private readonly appService: GrafanaLoggerService) {}

  @EventPattern('log')
  async handlelog({ data }: Record<string, any>) {
    this.appService.log({ ...data });

    // business logic
  }
  @EventPattern('error')
  async handleerror({ data }: Record<string, any>) {
    this.appService.error({ ...data });

    // business logic
  }
}
GrafanaLoggerService.ts

import {
  Injectable,
  Scope,
  Inject,
  Logger,
  InternalServerErrorException,
} from '@nestjs/common';
import { WINSTON_MODULE_PROVIDER } from 'nest-winston';
import { Logger as ILogger } from 'winston';
import winston from 'winston';
@Injectable({ scope: Scope.TRANSIENT })
export class GrafanaLoggerService extends Logger {
  constructor(
    @Inject(WINSTON_MODULE_PROVIDER) private readonly logger: ILogger,
  ) {
    super();
  }

  log({ lable, message, ...meta }) {
    this.logger.info({
      message: message,
      level: 'debug',
      labels: { name: lable },
      ...meta,
    });
  }

  error({ lable, message, ...meta }) {
    this.logger.error({
      message: message,
      level: 'error',
      labels: { name: lable },
      ...meta,
    });
  }
}

Now for actuall logging we will be using winston and winston-loki as transporter so our appmodule will look like

import { Module } from '@nestjs/common';
import { AppController } from './app.controller';

import { WinstonModule } from 'nest-winston';
import * as winston from 'winston';
const LokiTransport = require('winston-loki');
import { ConfigModule, ConfigService } from '@nestjs/config';
import { GrafanaLoggerService } from './app.service';

const {
  combine,
  timestamp,
  label,
  printf,
  prettyPrint,
  logstash,
  json,
  simple,
} = winston.format;
@Module({
  imports: [
    WinstonModule.forRootAsync({
      imports: [
        ConfigModule.forRoot({
          isGlobal: true,
        }),
      ],
      useFactory: (configService: ConfigService) => ({
        // options
        format: winston.format.combine(
          winston.format.colorize({
            all: true,
            level: true,
            colors: {
              info: 'green',
              error: 'red',
            },
          }),

          json(),
        ),
        transports: [
          new LokiTransport({
            host: 'http://lokiinstance:3100',
            json: true,
          }),
        ],
      }),
      inject: [ConfigService],
    }),
  ],
  controllers: [AppController],
  providers: [GrafanaLoggerService],
})
export class AppModule {}

once this is done we will create a docker image out of it and start the container in the same network

create a docker file

dockerfile



FROM node:14-alpine AS builder
WORKDIR /app
COPY ./package.json ./
RUN npm config set registry http://registry.npmjs.org/ 

RUN  npm install 
COPY . .
RUN npm run build

FROM node:14-alpine
WORKDIR /app
COPY --from=builder /app .
RUN ls
CMD ["npm","run", "start:prod"]

once you have the docker file in the root of your microservice

docker build -t logger-service .
docker run -d --network smartcodehubnetwork --name logger-service logger-service:latest
docker logs logger-service

Step 6 : Connect client to the microservice

Again ill be using nestjs to create a sample api project which we will connect to the logging service and emit some messages

inside your appmodule

 ...
 
 imports: [
    ClientsModule.register([
      {
        name: 'LOGGER_SERVICE',
        transport: Transport.NATS,
        options: {
          url: 'nats://nats-server:4222',
        },
      },
    ]),
  ],
  .....

once this is done inside the app component you can now  inject the service and use it as follows

import { Controller, Logger, LoggerService, Inject, Get } from '@nestjs/common';
import { LoggerService } from './InterfaceLayer/Logger/ILoggerService';
import { ClientProxy } from '@nestjs/microservices';

@Controller()
export class AppController {
  constructor(  @Inject('LOGGER_SERVICE') private client: ClientProxy) {
  }
  @Get()
  async getOne() {
      this.client.emit('log', {
          data: {
            lable: "Smartcodehub",
            message,
            ...requestData,
          },
        })

    return 'Hello';
  }
}

and hurrey you have a very simple yet very helpfull logging system setup correctly you can just refresh the api page which will emit the message and you can see it in real time

Thanks a lot for reading this article if it helps do let me know and do share your feedback if you have also faced such issues in your development experience