Lots of websocket pods restarts...💥

Trying to write a response after it has been sent. ☠️

  handle: function(request, response) {
      var requestUrl    = url.parse(request.url, true),
          requestMethod = request.method,
          self          = this;
  
      request.originalUrl = request.url;
  
      request.on('error', function(error) { self._returnError(response, error) });
      response.on('error', function(error) { self._returnError(null, error) });
  
      if (this._static.test(requestUrl.pathname))
        return this._static.call(request, response);
  
      // http://groups.google.com/group/faye-users/browse_thread/thread/4a01bb7d25d3636a
      if (requestMethod === 'OPTIONS' || request.headers['access-control-request-method'] === 'POST')
        return this._handleOptions(request, response);
  
      if (EventSource.isEventSource(request))
        return this.handleEventSource(request, response);
  
      if (requestMethod === 'GET')
        return this._callWithParams(request, response, requestUrl.query);
  
      if (requestMethod === 'POST')
        return this._concatStream(request, function(data) {
          var type   = (request.headers['content-type'] || '').split(';')[0],
              params = (type === 'application/json')
                     ? { message: data }
                     : querystring.parse(data);
  
          request.body = data;
          this._callWithParams(request, response, params);
        }, this);
  
      this._returnError(response, { message: 'Unrecognized request type' });
    },
  

   _callWithParams: function(request, response, params) {
        //...

        if (origin) {
          headers['Access-Control-Allow-Origin'] = origin;
        }
        headers['Cache-Control'] = 'no-cache, no-store';
  
        this._server.process(message, request, function(replies) {
          // ...
          headers['Content-Length'] = new Buffer(body, 'utf8').length.toString();
          headers['Connection'] = 'close';
  
          response.writeHead(200, headers);
          response.end(body);
        }, this);
    },
  

Why did it happen?

Deep dive

• web sdk opens long-polling connection, quickly closes it
• error handler on 114 fires, writes 400 response (ignored)
• subsequently, server tries to respond to long-polling connection and fails to write to response
• in node 16 → error handled on line 115
• in node 20 → error bubbles up differently for some reason; unhandled

"for some reason"

• This is a good view to take as a developer
• Things can happen for reasons you don't know
• Defensive programming
• NASA: The Power of Ten

"realtime"

What is real-time?

                      your answer here
  

• Event-driven - respond within time guarantee (ms)
• Often used in safety-critical apps (fly-by-wire, braking systems, etc)
• Time guarantees cannot allow system load to interfere with deadlines
  • Shed load to maintain time guarantees
  • New requests are shed so the system can honour requests it already accepted
  • Dropping older jobs → nothing ever gets processed

• multi-threaded - for parallelism
• concurrent - different parts of code simultaneously
  
  
  
  
  
  
  
  

event-driven

        request.on ('error', function ... );
        response.on ('error', function ... );
    

Concurrency in Node

• In async tasks triggered by thread pool
• No parallel execution of user code, but...
• There is concurrency of async tasks
• Event handling code can be interleaved

  bool strike (int weapon, int strength) {
    static int hitpoints = 100;
    int damage = getDamage(weapon, strength);
    bool rc = false;

    if (hitpoints && damage) {
      hitpoints = hitpoints - totalDamage;
      if (hitpoints < 0) hitpoints = 0;
      rc = true;
    }

    return rc;
  }
  

  async function handlerOne (req) {
    await blahblah();
    req.end(400, "your bad");
  }

  async function handlerTwo (req) {
    await yadayada();
    req.end(500, "my bad");
  }        
  

Take-aways

• Concurrency in node !== parallelism
• multi-threading is hidden in the thread pool (libuv)
• Be mindful of shared state in event-driven code
• Design for unexpected things "for some reason"
• (bonus) pure functions with no side effects may avoid critical sections

Credits

• Gabriel Périard
• Julian Garritano
• Alexandre Brun
• Mike Spensieri
• Alex "AJ" Sincennes
• #sunco-f-node-20-upgrade peeps

• Wikipedia on concurrency
• MIT on concurrency
• AWS on load shedding
• NASA's Power of Ten rules