In this competition, a classification algorithm is supposed to accurately identify which customers have the most potential business value for Red Hat based on their characteristics and activities.
For more information, please visit: https://www.kaggle.com/c/predicting-red-hat-business-value
In [2]:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
Loading the dataset
In [14]:
print("Read people.csv...")
people = pd.read_csv(r'..\RedHat\people.csv',
dtype={'people_id': np.str,
'activity_id': np.str,
'char_38': np.int32},
parse_dates=['date'])
In [15]:
print("Load train.csv...")
train = pd.read_csv(r'..\RedHat\act_train.csv',
dtype={'people_id': np.str,
'activity_id': np.str,
'outcome': np.int8},
parse_dates=['date'])
In [16]:
print("Load test.csv...")
test = pd.read_csv(r'..\RedHat\act_test.csv',
dtype={'people_id': np.str,
'activity_id': np.str},
parse_dates=['date'])
Review input features
In [6]:
print ("\n\n---------------------")
print ("TRAIN SET INFORMATION")
print ("---------------------")
print ("Shape of training set:", train.shape, "\n")
print ("Column Headers:", list(train.columns.values), "\n")
print (train.dtypes)
In [7]:
import re
missing_values = []
nonumeric_values = []
print ("TRAINING SET INFORMATION")
print ("========================\n")
for column in train:
# Find all the unique feature values
uniq = train[column].unique()
print ("'{}' has {} unique values" .format(column,uniq.size))
if (uniq.size > 10):
print("~~Listing up to 10 unique values~~")
print (uniq[0:10])
print ("\n-----------------------------------------------------------------------\n")
# Find features with missing values
if (True in pd.isnull(uniq)):
s = "{} has {} missing" .format(column, pd.isnull(train[column]).sum())
missing_values.append(s)
# Find features with non-numeric values
for i in range (1, np.prod(uniq.shape)):
if (re.match('nan', str(uniq[i]))):
break
if not (re.search('(^\d+\.?\d*$)|(^\d*\.?\d+$)', str(uniq[i]))):
nonumeric_values.append(column)
break
print ("\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n")
print ("Features with missing values:\n{}\n\n" .format(missing_values))
print ("Features with non-numeric values:\n{}" .format(nonumeric_values))
print ("\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n")
TRAINING SET INFORMATION ======================== 'people_id' has 151295 unique values ~~Listing up to 10 unique values~~ ['ppl_100' 'ppl_100002' 'ppl_100003' 'ppl_100006' 'ppl_100013' 'ppl_100019' 'ppl_100025' 'ppl_100028' 'ppl_100029' 'ppl_100032'] ----------------------------------------------------------------------- 'activity_id' has 2197291 unique values ~~Listing up to 10 unique values~~ ['act2_1734928' 'act2_2434093' 'act2_3404049' 'act2_3651215' 'act2_4109017' 'act2_898576' 'act2_1233489' 'act2_1623405' 'act2_1111598' 'act2_1177453'] ----------------------------------------------------------------------- .... .... ....
In [8]:
print ("\n\n---------------------")
print ("PEOPLE SET INFORMATION")
print ("---------------------")
print ("Shape of training set:", people.shape, "\n")
print ("Column Headers:", list(people.columns.values), "\n")
print (people.dtypes)
In [9]:
import re
missing_values = []
nonumeric_values = []
print ("PEOPLE SET INFORMATION")
print ("========================\n")
for column in people:
# Find all the unique feature values
uniq = people[column].unique()
print ("'{}' has {} unique values" .format(column,uniq.size))
if (uniq.size > 10):
print("~~Listing up to 10 unique values~~")
print (uniq[0:10])
print ("\n-----------------------------------------------------------------------\n")
# Find features with missing values
if (True in pd.isnull(uniq)):
s = "{} has {} missing" .format(column, pd.isnull(people[column]).sum())
missing_values.append(s)
# Find features with non-numeric values
for i in range (1, np.prod(uniq.shape)):
if (re.match('nan', str(uniq[i]))):
break
if not (re.search('(^\d+\.?\d*$)|(^\d*\.?\d+$)', str(uniq[i]))):
nonumeric_values.append(column)
break
print ("\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n")
print ("Features with missing values:\n{}\n\n" .format(missing_values))
print ("Features with non-numeric values:\n{}" .format(nonumeric_values))
print ("\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n")
PEOPLE SET INFORMATION ======================== 'people_id' has 189118 unique values ~~Listing up to 10 unique values~~ ['ppl_100' 'ppl_100002' 'ppl_100003' 'ppl_100004' 'ppl_100006' 'ppl_10001' 'ppl_100010' 'ppl_100013' 'ppl_100019' 'ppl_100025'] ----------------------------------------------------------------------- 'char_1' has 2 unique values ['type 2' 'type 1'] ----------------------------------------------------------------------- 'group_1' has 34224 unique values ~~Listing up to 10 unique values~~ ['group 17304' 'group 8688' 'group 33592' 'group 22593' 'group 6534' 'group 25417' 'group 4204' 'group 45749' 'group 36096' 'group 18035']
So we have 2197291 rows and 15 columns in train dataset and 189118 rows and 41 columns in people dataset.
There are no missing values. All variables are categorical, with the exception of ‘char_38′ in the people file, which is a continuous numerical variable. All the data is categorical except the outcome which contains yes and no . According to submission file , we only need activity_id and outcome. in the dataset, people_id’ has 151295 unique values and no missing values, therefore approximately 14.5 activity_id is associated with each person. There are 7 different types of activities. All corresponding activity characteristics have 2039676 missing values except char_10 . Each row in the activity file represents a unique activity performed by a person on a certain date.
Type 1 activities are different from type 2-7 activities because there are more known characteristics associated with type 1 activities (nine in total) than type 2-7 activities (which have only one associated characteristic).
Also, each activity has nan and type ‘int’. We will split the activities at type and use the unique integer as the numerical value.
Process the dataframe
In [17]:
print ("Processing...")
for table in [train, test]:
table['year'] = table['date'].dt.year
table['month'] = table['date'].dt.month
table['day'] = table['date'].dt.day
table.drop('date', axis=1, inplace=True)
table['activity_category'] = table['activity_category'].str.lstrip('type ').astype(np.int32)
for i in range(1, 11):
table['char_' + str(i)].fillna('type 0', inplace=True)
table['char_' + str(i)] = table['char_' + str(i)].str.lstrip('type ').astype(np.int32)
people['year'] = people['date'].dt.year
people['month'] = people['date'].dt.month
people['day'] = people['date'].dt.day
people.drop('date', axis=1, inplace=True)
people['group_1'] = people['group_1'].str.lstrip('group ').astype(np.int32)
for i in range(1, 10):
people['char_' + str(i)] = people['char_' + str(i)].str.lstrip('type ').astype(np.int32)
for i in range(10, 38):
people['char_' + str(i)] = people['char_' + str(i)].astype(np.int32)
According to kaggle, there is a suggestion to join the train and people files together into a single data set to develop a predictive model with this data. The two files can be joined together using person_id as the common key.
In [18]:
print("Merge...")
train = pd.merge(train, people, how='left', on='people_id', left_index=True)
train.fillna(0.0, inplace=True)
test = pd.merge(test, people, how='left', on='people_id', left_index=True)
test.fillna(0.0, inplace=True)
We simply drop the people_id attribute, since we do not expect it to be informative about the outcome status.
In [19]:
train = train.drop(['people_id'], axis=1)
In [20]:
train.head(2)
Out[20]:
Summary Statistics
In [21]:
train.describe()
Out[21]:
Process the dataset for Machine Learning
In [22]:
#Separate label and data
Y = train['outcome']
X = train.drop(['outcome'], axis=1)
Get the train data features, skipping the first column ‘activity_id’ as it is still object dtype()
In [23]:
X = X.iloc[:,1:]
PCA Visualization
In [32]:
from sklearn.decomposition import PCA, KernelPCA
components = 8
pca = PCA(n_components=components).fit(X)
In [33]:
#Show explained variance for each component
pca_variance_explained_df = pd.DataFrame({
"component": np.arange(1, components+1),
"variance_explained": pca.explained_variance_ratio_
})
ax = sns.barplot(x='component', y='variance_explained', data=pca_variance_explained_df)
ax.set_title("PCA - Variance explained")
plt.show()
In [51]:
sns.set_style('whitegrid')
X_pca = pd.DataFrame(pca.transform(X)[:,:2])
X_pca['target'] = Y.values
X_pca.columns = ["x", "y", "target"]
sns.lmplot('x','y',
data=X_pca,
hue="target",
fit_reg=False,
markers=["o", "x"],
palette="Set1",
size=7,
scatter_kws={"alpha": .2}
)
plt.show()
Evaluating Algorithms
Random Forest
In [55]:
#Create the random forest object:
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(n_estimators=100)
# Fit the model to our training data
rfc = rfc.fit(X, Y)
score = rfc.score(X, Y)
"Mean accuracy of Random Forest: {0}".format(score)
Out[55]:
Logistic Regression
In [54]:
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression(max_iter=100)
# Fit the model to our training data
lr = lr.fit(X, Y)
score = lr.score(X, Y)
"Mean accuracy of Logistic Regression: {0}".format(score)
Out[54]:
Linear Discriminant Analysis
In [57]:
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
lda = LinearDiscriminantAnalysis()
# Fit the model to our training data
lda = lda.fit(X, Y)
score = lda.score(X, Y)
"Mean accuracy of Linear Discriminant Analysis: {0}".format(score)
Out[57]:
Gradient Boosting Classifier
In [58]:
from sklearn.ensemble import GradientBoostingClassifier
gbc = GradientBoostingClassifier()
# Fit the model to our training data
gbc = gbc.fit(X, Y)
score = gbc.score(X, Y)
"Mean accuracy of GradientBoostingClassifier: {0}".format(score)
Out[58]:
Ada Boost Classifier
In [59]:
from sklearn.ensemble import AdaBoostClassifier
abc = AdaBoostClassifier()
# Fit the model to our training data
abc = abc.fit(X, Y)
score = abc.score(X, Y)
"Mean accuracy of AdaBoostClassifier: {0}".format(score)
Out[59]:
Analysis shows that Random Forest Classifier is the best for training with the accuracy of 1.
Prepare the test dataset for prediction
In [26]:
#drop the people_id
test = test.drop(['people_id'], axis=1)
In [27]:
# Get the test data features, skipping the first column 'PassengerId'
test_x = test.iloc[:, 1:]
In [29]:
# Predict the outcome values for the test data
test_y = list(map(int, rfc.predict(test_x)))
In [30]:
#file for submission
test['outcome'] = test_y
test[['activity_id', 'outcome']] \
.to_csv('results-rfc.csv', index=False)
