Divide and Conquer – Microservices with Node.js

26. Oct 2017

Más contenido relacionado


Divide and Conquer – Microservices with Node.js

  1. Divide and conquer Christian Steiner /
  2. Basti • Sebastian Springer • from Munich • work @ MaibornWolff GmbH • • @basti_springer • JavaScript Developer
  3. Agenda HTTP Microservice with Express • Routing • Structure • Database • Logging • Tests • Container Commandbased Microservices with Seneca • Patterns • Plugins • Queueing Rainer Sturm /
  4. Microservices Small Scaleable Resilient Independent Testable Focused
  5. JavaScript is nearly everywhere. There are a lot of OSS Modules. Node.js was built for the web Node.js supports most of the protocols, systems and databases. The platform is lightweight.
  6. Example A microservice that delivers articles for a web shop. M. Hermsdorf /
  7. Webshop Client API Articles Payment Users DB DB DB Auth Logger
  8. List of articles Router Controller Model Datenbank Logger
  9. Packages S. Hofschlaeger /
  10. Express.js Express is a lightweight (< 2MB) web application framework for Node.js. Express supports HTTP and HTTPS in version 1 and 2. Express handles routing within a web application for you. Additionally it provides you with a plugin interface (Middleware).
  11. Installation yarn init yarn add express
  12. Structure of a Microservice . ├── db │   ├── config.js │   └── db.sqlite3 ├── package.json ├── spec └── src    ├── config.js    ├── controller.js    ├── index.js    ├── logger.js    ├── model.js    └── router.js
  13. Entry point index.js
  14. Entry point const express = require('express'); const router = require('./router'); const app = express(); router(app); app.listen(8080, () => { console.log('Service is listening to http://localhost: 8080'); });
  15. Entry point In this file all global packages are included and the app is configured. Additional features should be put in separate files.
  16. Router router.js
  17. Router const controller = require('./controller'); module.exports = (app) => { app.get('/article', controller.getAll); app.get('/article/:id', controller.getOne);'/article', controller.create); app.put('/article', controller.update); app.delete('/article', controller.remove); }
  18. Router The router defines all available routes of a microservice. A route consists of a HTTP method and a path. By adding variables to routes they become more flexible. To keep the code of your router readable, you should put the routing callbacks in a separate file.
  19. Controller controller.js
  20. Controller const model = require('./model'); module.exports = { getAll(req, res) { res.json( model.getAll() ); }, getOne(req, res) {}, create(req, res) {}, update(req, res) {}, remove(req, res) {} }
  21. Controller The controller holds your routing callback functions. It extracts information from the requests and hands the information over to the model. To access variables of your routes, you can use the req.params object. To deal with information within the request body, you should install the body-parser middleware. It introduces the req.body object which represents the message body.
  22. Model model.js
  23. Model The model contains the business logic of your application. It also controls access to the database. Usually every microservice has its own database, which increases independence between services.
  24. Model Node.js supports all common databases such as OracleDB, MySQL, Redis, MongoDB. To access your database, you have to install a database driver first. yarn add sqlite3 To further simplify the access, you can use various ORMs or ODMs yarn add orm
  25. ORM An ORM handles CRUD operations for you. It already implements all common use cases. You just have to configure it with the structure of your database. The ORM also takes care of some security features such as escaping.
  26. ORM const express = require('express'); const orm = require('orm'); const router = require('./router'); const dbConfig = require('./db/config.js'); const {dbPath} = require('./config.js'); const app = express(); app.use(, dbConfig)); router(app); app.listen(8080, () => { console.log('Service is listening to http://localhost: 8080'); });
  27. ORM module.exports = { define(db, models, next) { models.articles = db.define('articles', { id: Number, title: String, price: Number }); next(); } }
  28. Model Besides encapsulating database communication your model takes care of additional tasks such as validation of user input and various calculations. Most of these operations are asynchronous. To provide a clean API you should think about using promises, async functions or streams instead of plain callbacks.
  29. async getAll(req, res) { try { const articles = await model.getAll(req); res.json(articles); } catch (e) { res.status(500).send(e); } } getAll(req) { return new Promise((resolve, reject) => { req.models.articles.all((err, results) => { if (err) { reject(err); } else { resolve(results); } }); }); } controller model
  30. Logging Tim Reckmann /
  31. Logging Within your microservice errors might occur at any time. Of course you should be prepared to catch and handle them. But you should also keep a log of errors somewhere. A logger should not be a fixed part of a certain microservice. logging is a shared service which is available for all services in your application. Centralised logging provides some advantages over local logging such as scalability and an improved maintainability.
  32. Logging You can use log4js for remote logging in your application. This library provides a plugin interface for external appenders which support various log targets such as files or logstash. For remote logging you could use the logstash appender and a centralised logstash server.
  33. Logging const log4js = require('log4js'); log4js.configure({ appenders: [{ "host": "", "port": 10001, "type": "logstashUDP", "logType": "database", "layout": { "type": "pattern", "pattern": "%m" }, "category": "database" }], categories: { default: { appenders: ['database'], level: 'error' } } }); module.exports = log4js;
  34. Logging const log4js = require('./logger'); module.exports = { getAll(req) { return new Promise((resolve, reject) => { req.models.articles.all((err, results) => { if (err) { reject(err); log4js.getLogger('database').error(err); } else { resolve(results); } }); }); } }
  35. Tests Dieter Schütz /
  36. Tests Quality and stability are two very important aspects of microservices. To ensure this, you need the ability to automatically test your services. There are two levels of tests for your microservice application: unittests for smaller units of code and integration tests for external interfaces. For unittests you can use Jasmine as a framework. With Frisby.js you can test your interfaces.
  37. Unittests yarn add jasmine node_modules/.bin/jasmine init
  38. Unittests const model = require('../model'); describe('model', () => { it('should handle a database error correctly', () => { const req = { models: { articles: { all: (cb) => {cb('error', null);} } } } model.getAll(req).catch((e) => { expect(e).toBe('error'); }); }) });
  39. Unittests To execute your tests, just issue the command npx jasmine. You have two variants for organising your tests. You can either store them in a separate directory, usually named “spec” or you put your tests where your source code is located.
  40. Mockery In your microservice application you have to deal with a lot of dependencies. They are resolved via the node module system. Mockery is a tool that helps you with dealing with dependencies. Mockery replaces libraries with test doubles. yarn add -D mockery
  41. Mockery const mockery = require('mockery'); beforeEach(() => { mockery.enable(); const fsMock = { stat: function (path, cb) {...} }; mockery.registerMock('fs', fsMock); }); afterEach(() => { mockery.disable(); });
  42. Integration tests Frisby.js is a library for testing REST interfaces. Frisby is an extension for Jasmine. So you don’t have to learn an additonal technology. In order to run Frisby.js, you have to install jasmine-node. yarn add -D frisby jasmine-node
  43. Integration tests require('jasmine-core'); var frisby = require('frisby'); frisby.create('Get all the articles') .get('http://localhost:8080/article') .expectStatus(200) .expectHeaderContains('content-type', 'application/json') .expectJSON('0', { id: function (val) { expect(val).toBe(1);}, title: 'Mannesmann Schlosserhammer', price: 7 }) .toss();
  44. PM2 Node.js is single threaded. Basically that’s not a problem because of its nonblocking I/O nature. To optimally use all available resources, you can use the child_process module. A more convenient way of locally scaling your application is using PM2. yarn add pm2
  45. PM2 pm2 start app.js pm2 start app.js -i 4 pm2 reload all pm2 scale <app-name> 10 pm2 list pm2 stop pm2 delete
  46. Docker All your services run in independent, self contained containers. You can start as many instances of a certain container as you need. To extend the features of your application, you simply add additional services by starting containers.
  47. Dockerfile FROM node:7.10 # Create app directory RUN mkdir -p /usr/src/app WORKDIR /usr/src/app # Install app dependencies COPY package.json /usr/src/app/ RUN yarn install # Bundle app source COPY . /usr/src/app EXPOSE 8080 CMD [ "yarn", "start" ]
  48. Docker docker build -t basti/microservice . docker run -p 8080:8080 -d basti/microservice
  49. API Gateway Jürgen Reitböck /
  50. API Gateway Each service of your application has to focus on its purpose. In order to accomplish this, you have to centralise certain services. A typical example for a central service is authentication. An API Gateway forwards authorised requests to the services of your application, receives the answers and forwards them to the client.
  51. Seneca Seneca follows a completely different approach. All services of your application communicate via messages and become independent of the transport layer.
  52. Installation yarn add seneca
  53. Service Definition const seneca = require('seneca')(); seneca.add({role: 'math', cmd: 'sum'}, controller.getAll); The first argument of the add method is the pattern, which describes the service. You are free to define the pattern as you want. A common best practice is to define a role and a command. The second argument is an action, a function that handles incoming requests.
  54. Service Handler async getAll(msg, reply) { try { const articles = await model.getAll(req); reply(null, JSON.stringify(articles)); } catch (e) { reply(e); } } Similar to express a service handler receives the representation of a request. It also gets a reply function. To create a response, you call the reply function with an error object and the response body.
  55. Service Call seneca.act({role: 'article', cmd: 'get'}, (err, result) => { if (err) { return console.error(err); } console.log(result); }); With seneca.act you consume a microservice. You supply the object representation of a message and a callback function. With the object, you trigger the service. As soon as the answer of the service is available, the callback is executed.
  56. Patterns Helmut / pixelio.deHelmut /
  57. Patterns You can create multiple patterns of the same type. If a service call matches multiple patterns, the most specific is used. You can use this behaviour to implement versioning of interfaces.
  58. Plugins Klicker /
  59. Plugins A plugin is a collection of patterns. There are multiple sources of plugins: built-in, your own plugins and 3rd party plugins. Plugins are used for logging or debugging.
  60. Plugins You organise your patterns with the use method. The name of the function is used for logging purposes. You can pass options to your plugin to further configure it. The init pattern is used instead of a constructor. const seneca = require('seneca')(); function articles(options) { this.add({role:'article',cmd:'get'}, controller.getAll); } seneca.use(articles);
  61. Plugins const seneca = require('seneca')(); function articles(options) { this.add({role:'article',cmd:'get'}, controller.getAll); this.wrap({role:'article'}, controller.verify); } seneca.use(articles); The wrap method defines features that are used by multiple patterns. With this.prior(msg, respond) the original service can be called.
  62. Client/Server cre8tive /
  63. Server function articles(options) { this.add({role:'article',cmd:'get'}, controller.getAll); this.wrap({role:'article'}, controller.verify); } require('seneca')() .use(articles) .listen(8080) Just like in the ordinary web server the listen method binds the server to the port 8080. Your service can then be called by the browser or another server: http://localhost:8080/act?role=article&cmd=get
  64. Client require('seneca')() .client(8080) .act({role: 'article', cmd: 'get'}, console.log); The client method lets you connect to a seneca microservice.
  65. Change the transport // client seneca.client({ type: 'tcp', port: 8080 }); // server seneca.listen({ type: 'tcp', port: 8080 }); By providing the type ‘tcp’ the TCP protocol is used instead of HTTP as a communication protocol.
  66. Integration in Express
  67. Integration in Express const SenecaWeb = require('seneca-web'); const Express = require('express'); const router = new Express.Router(); const seneca = require('seneca')(); const app = Express(); const senecaWebConfig = { context: router, adapter: require('seneca-web-adapter-express'), options: { parseBody: false } } app.use( require('body-parser').json() ) .use( router ) .listen(8080); seneca.use(SenecaWeb, senecaWebConfig ) .use('api') .client( { type:'tcp', pin:'role:article' } );
  68. Integration in Express With this configuration seneca adds routes to your Express application. You have to adopt your seneca patterns a little bit. All routes defined with seneca.act('role:web', {routes: routes}) are added as Express routes. Via the path-pattern a corresponding matching is done.
  69. Communication via a queue S. Hofschlaeger /
  70. Queue You can use various 3rd party plugins to communicate via a message queue instead of sending messages over network. The main advantage of using a Queue is decoupling client and server. Your system becomes more robust.
  71. Queue yarn add seneca-servicebus-transport
  72. Queue require('seneca')() .use('seneca-servicebus-transport') .use(articles) .listen({ type: 'servicebus', connection_string: '...' }); require('seneca')() .use('seneca-servicebus-transport') .client({ type: 'servicebus', connection_string: '...' }) .act({role: 'article', cmd: 'get'}, console.log); Server Client
  73. Microservices, the silver bullet? Microservices might be a good solution for certain problems but definitely not for all problems in web development. Microservices introduce an increased complexity to your application. There is a lot of Node.js packages dealing with this problem. Some of these packages are outdated or of bad quality. Take a look at GitHub statistics and if it fits your needs.
  74. Questions Rainer Sturm /
  75. CONTACT Sebastian Springer MaibornWolff GmbH Theresienhöhe 13 80339 München @basti_springer