Overview

Python at the time of writing is the defacto language for prototyping and developing machine learning algorithms. In this post, we will be using Scala to develop a simple linear regressor model. We will do this with the help of the Scala numerics library Breeze.

Machine Learning with Scala Linear Regression

As it is well known, the linear regression model assumes the following functional form for the predictor $\hat{y}$

$$\hat{y}_i = a x_i + b$$

The loss function has the following form

$$L(\mathbf{w}) = \sum_{i}^N (y_i - \hat{y}_i)^2 = \sum_{i}^N (y_i - (a x_i + b))^2$$

where $\mathbf{w}$ is the parameters coefficients with $\mathbf{w} = [a, b]$. The gradient of the loss function with respect to the parameters is as follows

$$\frac{\partial L}{\partial a} = -2 \sum_{i}^N (y_i - \hat{y}_i) x_i$$

$$\frac{\partial L}{\partial b} = -2 \sum_{i}^N (y_i - \hat{y}_i)$$

The term

$$SSE = \sum_{i}^N (y_i - \hat{y}_i)^2$$

is called the sum of squared errors or SSE. If we divide with the number of training examples, $N$, then we get the so-called mean squared error or MSE

$$MSE = \frac{1}{N}\sum_{i}^N (y_i - \hat{y}_i)^2$$

We first import some useful packages

import breeze.linalg._
import breeze.optimize.{DiffFunction, minimize}

We wrap the loss function and its gradient calculation into an object class

object LinearRegression
{
  def L(x: DenseMatrix[Double], y: DenseVector[Double], parameters: DenseVector[Double]): Double = {
    val yHat = x * parameters
    var value = 0.0
    for( i <- 0 until yHat.size){
      val diff =  y(i) - yHat(i)
      value += diff * diff
    }
    value
  }


  def gradL(x: DenseMatrix[Double], y: DenseVector[Double], 
            parameters: DenseVector[Double]): DenseVector[Double]={

    val yHat = x * parameters

    // we have as many components as columns
    val gradients = DenseVector.zeros[Double](x.cols)

    for( i <- 0 until yHat.size){
      var diff =  y(i) - yHat(i)

      for( c <- 0 until gradients.size){
         diff *= x(i, c)
         gradients(c) += diff
      }
    }
    -2.0 * gradients
  }
}

This is the class that wraps the linear regression model.


class LinearRegression{

  // The model parameters
  var parameters: DenseVector[Double] = null

  // Flag indicating if the interception term is used
  var useIntecept: Boolean=true;

  // constructor
  def this(numFeatures: Int, useIntercept: Boolean=true){
    this()
    init(numFeatures = numFeatures, useIntercept = useIntercept)
  }


  // train the model
  def train(x: DenseMatrix[Double], y: DenseVector[Double])={

    // set up the optimization
    val f = new DiffFunction[DenseVector[Double]] {
      def calculate(parameters: DenseVector[Double]) = (LinearRegression.L(x, y, parameters=parameters),
        LinearRegression.gradL(x, y, parameters = parameters))
    }

    this.parameters = minimize(f, this.parameters)

  }

  // the initialization function
  def init(numFeatures: Int, useIntercept: Boolean=true): Unit = {

    val totalFeatures = if(useIntercept) numFeatures + 1 else numFeatures
    this.parameters = DenseVector.zeros[Double](totalFeatures)
    this.useIntecept = useIntercept
  }

}

Let's put this into action with a simple example.

object LinearRegressionExe_1 {

  def main(args: Array[String]):Unit={

    // data set
    val x = LineDataSetLoader.lineSplit(0.0, 10.0, 100)

    System.out.println("Number of training examples: " + x.size)

    val coeffs = Array[Double](1.0, 2.0)
    val poly = new Polynomial(coeffs)
    val y = poly.values(x)



    // the feature matrix
    val featureMatrix = DenseMatrix.horzcat(DenseMatrix.ones[Double](x.size, 1), x.toDenseMatrix.t)

    // model
    val model = new LinearRegression(numFeatures = 1, useIntercept = true)

    model.train(x=featureMatrix,y=y)

    println(s"Polynomial coeffs ${poly.getCoeffsAsDenseVector}")
    println(s"Linear regressor coeffs ${model.getParameters}")
  }

}

You can find the complete example in this repo.

Summary

In this post we looked into how to develop a simple linear regression model with Scala. The Scala numerics library Breeze greatly simplifies the development.

References

Linear regression
Pascal Bugnion, Patric R. Nicolas, Alex Kozlov, Scala: Applied Machine Learning