# Melbourne House Price Prediction using XGBoost

In the previous article we used XGBoost for classification problem on the titanic dataset, in this article we will do regression on Melbourne Housing Dataset using the Extreme Gradient Boosting from the Caret Package available in R. We will predict the prices of houses and will try to get the best accuracy by avoiding over fitting issues.

We have no information regarding the variables of the dataset but you can find out easily by searching on Google. Most of the variables are easy to understand.

### Download and load Dataset

The first step is to download the dataset from Kaggle or Github. The housing dataset comes in variety of forms, you may find housing dataset having more than `70`

variables, but this dataset is quite simple and analysis is performed for the sake of learning.

`train <- read.csv('Aus_housing.csv', stringsAsFactors =FALSE)`

### Install and Load Libraries

We will need to install many libraries and then load them into R.

```
#install and load required packages
install.packages(c("gridExtra","ggplot2","e1071", "caret", "doSNOW", "ipred", "xgboost", "dplyr"))
library(e1071)
library(caret)
library(doSNOW)
library(ipred)
library(xgboost)
library(dplyr)
library(ggplot2)
library(gridExtra)
```

### Clean Dataset

Before we analyze the data we need to clean the dataset. Cleaning process takes a lot of time, we have to remove duplicate records and missing value records. We can see by viewing the dataset that it contains many missing values, we can remove them.

The outcome variable *Price* contains lot of missing values we can remove them (We can also store them separately and predict once our model is ready). Similarly, we can see many attributes values missing in each rows, we can remove all of them with the following:

```
#removing missing records based on Price and Bedroom2
train <- train[!is.na(train$Price),]
train <- train[!is.na(train$Bedroom2),]
```

You will still see many missing values like * BuildingArea *but they are now in only few columns we can impute them later.

Also, * BuildingArea* and

*contain many zero values, these are actually missing values we can replace them with*

`LandSize`

`NA`

.```
# replacing 0 with NA
train$BuildingArea[train$BuildingArea == 0] <- NA
train$Landsize[train$Landsize == 0] <- NA
```

### Data Analysis

Now comes the data analysis step where we will explore data, also known as EDA (Exploratory Data Analysis). We need to analyze every variable individually and with the outcome variable but here I will analyze only important variables.

The first step we will plot a histogram of the outcome variable just to know how the data is distributed.

`# For plotting continious individual plotsggplot(aes(Price),data=train)+geom_histogram() `

Similarly, seeing distribution of * YearBuilt*. This graph seems to show that the majority of houses are from the the mid 20th century.

`ggplot(aes(YearBuilt),data=train)+ geom_histogram()`

Nowmultiple histograms to see distribution.

```
# Multiple Plots
a <- ggplot(aes(Rooms),data=train)+ geom_bar()
b <- ggplot(aes(Bedroom2),data=subset(train, Bedroom2 < 20))+
geom_bar()+ scale_x_discrete(breaks=seq(0,8,2), limits= c(0:8))
c <- ggplot(aes(Bathroom),data=train)+ geom_bar()
d <- ggplot(aes(Car),data=train)+ geom_bar()
grid.arrange(a, b, c, d, ncol=2, nrow=2)
```

`Plotting`

* Date *vs

*, the plot does not give enough information, the*

`Price`

*Price*seems to be independent of the

*Date*. It would have worked great if the rangewas atleast five years.

`# Date plotggplot(aes(as.Date(Date,'%d/%m/%Y'),Price), data= train)+ geom_point()`

Plotting multiple graphs for `Landsize`

` vs `

* Price* and

`BuildingArea`

` vs `

*with respect to*

`Price`

`Type`

` and `

*. It will tell how the Price ischanging with the change in other variables.*

`Method`

```
# Multi plot
w <- ggplot(aes(Landsize,Price), data= train)+
geom_point(alpha=1/3, aes(color= Type))+
coord_cartesian(xlim = 0:10000)
x <- ggplot(aes(Landsize,Price), data= train)+
geom_point(alpha=1/3, aes(color= Method))+
coord_cartesian(xlim = 0:10000)
y <- ggplot(aes(BuildingArea,Price), data= train)+
geom_point(alpha=1/5, aes(color= Type))+
coord_cartesian(xlim = 0:1000, ylim = 0:5e+06)
z <- ggplot(aes(BuildingArea,Price), data= train)+
geom_point(alpha=1/5, aes(color= Method))+
coord_cartesian(xlim = 0:1000, ylim = 0:5e+06)
grid.arrange(w, x, y, z, ncol=2, nrow=2)
```

If we look at the first column, the second plot (`*BuildingArea *vs *Price *vs *Type`*) is far more meaningful than the first one. The * BuildingArea *and

*both are showing how effective they could be in predicting the*

`Type`

*.*

`Price`

`PlottingLattitude `

`vs `

`Longitude `

`vs `

*, to see how Council Areas are spread in the form of map.*

`CouncilArea`

`ggplot(aes(Lattitude,Longitude), data= train)+ geom_point(alpha=1/5,aes(color= CouncilArea))`

Similarly, you can create your own visualization and interpret them in a different way. These visualization will be helpful in feature selection and dimensionality reduction. I would recommend you to dive deep and do more EDA.

### Feature Selection

Now its time to reduce the number of features, in other words reduce the dimensions. There are several irrelevant features that we can directly filter out and then use a feature subset selection approach to find one since the number of variables are not that high.

Variables like `Suburb`

`, `

`Address`

` and `

* SellerG* are categorical and have many categories, including this would not improve the model and further the model will take more time in training.

*Postcode*is also not relevant because that is more like a categorical variable. (Visualize them for better understanding)

I did not get the purpose of * Distance *variable in the dataset, even their is no such meaningful relation between

*and*

`Distance`

*, so I neglect it out. (Dig Deep!)*

`Price`

* Landsize *and

*seem to be a common feature in predicting the prices, but in this case its not. Even after trying out with them, the model seems to start overfitting. (See the Visualization)*

`Date`

Now selecting the features, from the visualization presented above and some commonsense, we can come up with the following features:

```
# Feature selection
train <- subset(train, select = c(Rooms, Bedroom2, Bathroom, Car, Type,
Method, BuildingArea, YearBuilt, CouncilArea,
Lattitude, Longitude, Price))
```

### Impute Missing Values

In order to impute missing values we need to create dummy variables but before that if we have any non-missing categorical variable that has many categories we can remove them and add them back once all missing values are computed.

In our case we have `CouncilArea`

**, **so storing it in a temporary dataframe and then removing it. This will help us to impute the missing values quickly.

```
# For back up of values
temp = data.frame(train$CouncilArea)
train <- subset(train, select = -c(CouncilArea))
```

Now using `Caret`

package to create dummy variables, then impute them and finally replace the imputed variables in the original dataset.

```
# Transform all feature to dummy variables.
dummy.vars <- dummyVars(~ ., data = train[, -1])
train.dummy <- predict(dummy.vars, train[, -1])
View(train.dummy)
# Now, impute!
pre.process <- preProcess(train.dummy, method = "bagImpute")
imputed.data <- predict(pre.process, train.dummy)
View(imputed.data)
#Assignign imputed values to original dataset
train$BuildingArea <- imputed.data[, 12]
train$YearBuilt <- imputed.data[, 13]
train$Lattitude <- imputed.data[, 14]
train$Longitude <- imputed.data[, 15]
#Adding back the CouncilArea
train$CouncilArea <- temp$train.CouncilArea
```

### Split Data and Train Model

The next few steps are pretty easy, all you have to do is to setup stuff for the model, no need for deep explanation. The code is similar to what we did in the Titanic Dataset.

```
# Using 10-fold cross validation repeated 3 times
# Using a grid search for optimal model hyperparamter values.
train.control <- trainControl(method = "repeatedcv",
number = 10,
repeats = 3,
search = "grid")
# Leverage a grid search of hyperparameters for xgboost
tune.grid <- expand.grid(eta = c(0.05, 0.075, 0.1),
nrounds = c(50, 75, 100),
max_depth = 6:8,
min_child_weight = c(2.0, 2.25, 2.5),
colsample_bytree = c(0.3, 0.4, 0.5),
gamma = 0,
subsample = 1)
# For Parallel Processing use doSNOW
# Tune the number of thread or core based on your PC
cl <- makeCluster(4, type = "SOCK")
# Register cluster so that caret will know to train in parallel.
registerDoSNOW(cl)
caret.cv <- train(Price ~ .,
data = housing.train,
method = "xgbTree",
tuneGrid = tune.grid,
trControl = train.control)
stopCluster(cl)
# Examine caret's processing results
caret.cv
```

The last line above will show how the model performed based on different combination of hyperparameters, and will pick the best one.

### Test and Evaluate Model

The model is ready, we need to test it on the test dataset which we obtained during the splitting process. For continuous output variable, we can evaluate the model using Root Mean Squared Error, R-Squared, Minimum and Maximum Accuracy and Mean Absolute Percentage Error (MAPE).

```
# Making Predictions
preds <- predict(caret.cv, housing.test)
#Evaluation using RMSE, R-Squared
postResample(preds,housing.test$Price)
# Evaluation using min max accuracy and MAPE
actuals_preds <- data.frame(cbind(actuals=housing.test$Price, predicteds=preds))
head(actuals_preds)
min_max_accuracy <- mean(apply(actuals_preds, 1, min) / apply(actuals_preds, 1, max))
mape <- mean(abs((actuals_preds$predicteds - actuals_preds$actuals))/actuals_preds$actuals)
```

Similarly, we can reuse the above code for training dataset so that we can see how well our model has predicted on the test dataset. Also, we can see in different settings of the features whether there is an overfitting or not. If more variables are added in the feature list, then you would see an increase in training accuracy but slight decrease in test accuracy, meaning that the model is start to overfit.

### Output

**Training Dataset**

**Testing Dataset**

**Ideal Values**

**RMSE**

1.523547e+05

2.730161e+05

0

**R-Squared**

9.474203e-01

8.274691e-01

1

**M_M_Accuracy**

0.9024803

0.8723883

1

**MAPE**

0.0977454

0.1545473

0

The `M_M_Accuracy`

is quite well in this scenario, the reason for that is the * Price *contains big numbers so a

`10,000`

difference in prediction would show a big decrease in the accuracy, but in reality it will not matter that much when house price is `1,000,000`

.`RMSE` in our case is quite high considering the exponential term, the reason again is the big numbers in

*Price*. So neglect the exponential, then it becomes quite good and meaningful.

To improve the accuracy, feature engineering is now required. I created few features from * Address* and

*variables but that didn’t help, will update this article in case if I come up with better result.*

`YearBuilt`

You can find the full R code at the Github Repo.

For any question, feedback or suggestion, feel free to comment below.