Month: June 2014

After the 33rd Degree 2014 – main tools/techniques to trial or adopt

Every conference I have attended so far left me with new ideas and motivation to do things better on daily basis. This year 33rd Degree has given me the same boost as the previous ones.

This blog post is a quick overview of tools/techniques I am going to investigate deeper after the conference and try to introduce at least some of them at my daily work.

  • Project Lombok
    • boilerplate code generator
    • some features:
      • @NonNull annotation – generates null check for you
      • @Cleanup annotation – to ensure that a given resource is automatically cleaned up
      • @Getter / @Setter annotations- for every annotated field generates standard getter/setter methods
      • @Log annotation – for every class with the annotation generates logger field
  •  Logback
    • I have used it in one project but forget about it and stick to log4j
    • about 10 times faster than log4j
    • automatic configuration files reloading
    • automatic old log files compression and removal
    • maybe wait for Apache Log4j 2 ?
  • Logstash
    • a brilliant tool for managing application logs
    • provides web interface
  • vert.x
    • light, robust and high performance application platform for JVM
    • simple asynchronous API similar to the node.js one
    • project inspired by node.js so both have many similarities
    • components can be developed in different languages: Java, JavaScript, Groovy, …
    • scalability and high availability support out of the box
  • application metrics
    • start providing application metrics
    • a nice tool from codahale
    • already implemented it in Java application – fulfils all expectations
    • jooq
      • lets build typesafe SQL queries
      • fluent API
      • code completion for SQL
      • bugs found on compilation time
      • paid for use with non open source databases
    • Spring Boot
      • another excellent tool from Pivotal/SpringSource
      • quick application bootstrap
      • embeds Tomcat or Jetty
      • features such as metrics or health checks provided out of the box
      • no code generation
    • Modern Java applications deployment
      • awesome quote from +Axel Fontaine : “Running servers in production should be like going backpacking. You take the bare minimum with you. Anything else is going to hurt”
      • environment promotion instead of setting up a clean environment on development/test/production – that idea is not new for me. I have already seen it and use it on the Mainframe infrastructure.
      • embedded containers
    • Nashorn
      • JavaScript engine for JVM
      • amazing two-way integration with Java
    • walkmod
      • a tool to enforce consistent code conventions among project contributors
        • code licence snippets
        • code formatting
        • automatic refactoring – but I would think deeper before using it

    Measure and find bottlenecks before they affect your users 1/2

    Inspired by a few talks during the last 33rd Degree conference I decided to implement, in one of the applications I develop, metrics which allow developers or operation team monitor running application and possibly detect potential problems early on.

    After a quick investigation I decided to use the Metrics framework. What I expected was exposing at least some statistics concerning usage of application components. First of all I would use them during application stress tests to find particular components slowing down the application. After going into production, I imagine that such statistics would be helpful to monitor the running application.

    Metrics framework perfectly fits my expectations. It is a Java framework. What is more there are Javascript ports which I am going to use to monitor Node.js server (more about it in one of the next posts).
    I decided to integrate the tool with Spring application context using metrics-spring but of course it is possible to use it without Spring.

    Here is the Spring application context with Metrics support:

    <?xml version="1.0" encoding="UTF-8"?>
<beans
    xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:context="http://www.springframework.org/schema/context"
    xmlns:mvc="http://www.springframework.org/schema/mvc"
    xmlns:metrics="http://www.ryantenney.com/schema/metrics"   xsi:schemaLocation="http://www.springframework.org/schema/beans    http://www.springframework.org/schema/beans/spring-beans-3.2.xsd    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd    http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc.xsd    http://www.ryantenney.com/schema/metrics http://www.ryantenney.com/schema/metrics/metrics-3.0.xsd">
    <metrics:metric-registry id="metrics"/>
    <metrics:annotation-driven metric-registry="metrics"/>
    <metrics:reporter id="metricsJmxReporter" metric-registry="metrics" type="jmx"/>
    <metrics:reporter id="metricsLogReporter" metric-registry="metrics" type="slf4j" period="1m"/>
</beans>

    The configuration defining a few beans:

    • metrics-registry is the bean used to register generated metrics; its explicit definition is optional, if not defined new MetricRegistry bean is created
    • annotation-driven element tells that annotations are used to mark methods/beans under monitoring
    • reporter element is used to report gathered statistics to the defined consumers; there are a few reporter implementation provided (jmx, console, slf4j, ganglia, graphite); I decided to use two of them:
      • jmx (JmxReporter) exposing metrics as JMX MBeans; they can be explored using standard tools like jconsole or VisualVM
      • slf4j (Slf4jReporter) logging metrics to an SLF4J logger; period attribute defines the interval used to report statistics to a log file

    When configuration is done, it is time to annotate the bean methods which are to be monitored. To do that there is a simple @Timed annotation provided:

    @RequestMapping(value = "/transaction", method = RequestMethod.POST)
@ResponseBody
@Timed
public HttpEntity<SubmitResultDTO> transaction(@RequestBody NewTransactionDTO transaction) { ...}
    Using that simple configuration you get JMX MBeans exposed providing a nice set of metrics:
     sc1
    What is more, if any statistic is clicked an online chart is presented:

     

    sc2

    Besides the JMX reporter there is also the SLF4J reported defined, which logs the following pieces of information:

    sc3

     

    Except JMX or SLF4J reporting more sophisticated tools can be used to consume statistics provided my Metrics. I would recommend trying Ganglia or Graphite as there are reporters provided for those consumers (GangliaReporter and GraphiteReporter).

    Manage Node.js dependencies properly

    Author of every piece of software bigger than simple “Hello World” sooner or later will face problem with dependency management. In this post I would like to consider that problem and possible solutions in regard to Node.js applications.
    In Node.js world managing dependencies with npm is exceptionally simple, however there are some caveats which everyone should be aware of.

    Have a look at example package.json file:

    {  
  "name": "",  
  "description": "",  
  "version": "0.0.1",  
  "private": true,  
  "dependencies": {    
    "express": "~4.2.x",    
    "compression": "~1.0.x",    
    "serve-favicon": "~2.0.x",    
    "morgan": "~1.0.x",    
    "cookie-parser": "~1.1.x",    
    "body-parser": "~1.2.x",    
    "method-override": "~1.0.x",    
    "express-session": "~1.1.x",    
    "serve-static": "~1.1.x",    
    "connect-redis": "~2.0.x",    
    "redis": "~0.10.2",    
    "open": "~0.0.x",    
    "request": "~2.34.x",    
    "connect": "~2.14.x",    
    "connect-flash": "~0.1.x",    
    "crypto": "~0.0.x",    
    "passport": "~0.2.x",    
    "passport-local": "~1.0.x",    
    "underscore": "~1.6.x",    
    "async": "~0.6.x",    
    "moment": "~2.5.x",    
    "ejs": "~1.0.x",    
    "cookie": "~0.1.x",    
    "winston": "~0.7.x",    
    "path": "~0.4.x",    
    "stompjs": "~2.3.x",    
    "socket.io": "~0.9.x",    
    "forever-monitor": "~1.2.x",  
  },  

  "devDependencies": {    
    "supertest": "latest",    
    "should": "latest",    
    "karma": "latest",    
    "karma-junit-reporter": "latest",    
    "karma-ng-html2js-preprocessor": "latest",    
    "grunt": "latest",    
    "grunt-contrib-jshint": "latest",    
    "jshint-junit-reporter": "latest",    
    "grunt-devperf": "latest"  
    }
  }

    Every package has a version number which comply with what is described as semantic versioning. Generally speaking, it takes the form of major.minor.patch where major, minor and patch are integers increasing after each new release. Having package.json as above on first sight seems to be defensive approach to dependency management. We decide to every run of npm install download and install latest release of specified major.minor version. In other worlds, we accept bugfixes to the current version but refuse to download new minor or major versions. As long as it accepted on development environment, this approach needs to be reviewed when production deployment is considered. Deploying on production we want to be a 100% sure that we deploy software package which has been already tested. In that solution we cannot assure that. Running npm install on production will probably download new version (bugfix) of some package.

    How to deal with it?There are at least a few possible solutions:

    • use specific version (for example “forever-monitor”: “1.2.3”) in package.json
      • pros
        • every time npm install is executed the same version is downloaded and installed
        • every version change needs editing package.json file
      • cons
        • it is still possible that package author republishes an existing version of the package and as a result we get a different code on production
        • on development environment it is accepted that new patches are installed since the software is instantly tested
    • npm install dependencies on development environment and commit the results into version controlsystem
      • pros
        • we deploy the exact bits which were checked in into version control
      • cons
        • in specific cases npm install not only downloads packages but also builds system-dependent binaries so it is obvious that packages installed on Microsoft Windows system will be suitable for Linux operating system
        • it is error prone since someone can check in a source change but not regenerate the binaries
        • it is redundant since binaries can always be built from sources; as we do not commit built artifacts in Java ecosystem, we should not do that in Node.js environment too
    • npm shrinkwrap – the command locks down the versions of package’s dependencies; produces npm-shrinkwrap.json containg specific version at the current time which is used instead of package.json in subsequent npm install commands
      • pros
        • all dependencies are locked down so every npm install will download the same version
      • cons
        • there is still possibility that package author will republish code while not changing version number
    • npm shrinkwrap on development environment and tar up dependencies on test environment before distributing them to production; test development is the same as production as far as platform is considered
      • pros
        • changes to packages versions are acceptable on development environment (update shrinkwrapped package as described in manual)
        • version deployed on production is always the same version which was tested
      • cons
        • patches are not automatically downloaded every npm install on development environment and require some manual actions, but not in package.json file
    This was a quick review of a few available package management methods. Of course there are more possibilities such as maintaining a mirror of the part of npm registry but this a subject for another post. 
    Do not expect to find one, always suitable solution. If it is fine by you to specify exact package version you can carry on with it. Personally I prefer the last route (npm shrinkwrap and tar up dependencies).