Scikit-pandas、クロスバルーンスコアの特徴数

Question

私はKaggleのタイタニックチュートリアルを使ってscikitとそのパンダの統合に精通しています。私は自分のデータを整理して予測をしたいと思っています。私はパイプラインの適合と変換を呼び出すことができます - 残念ながら、cross_val_scoreで同じことをしようとするとエラーが発生します。私はsklearn-パンダを使用しています

が

をcross_val_scoreコードは以下の通りである：

mapping = [ 
     ('Age', None), 
     ('Embarked',LabelBinarizer()), 
     ('Fare',None), 
     ('Pclass',LabelBinarizer()), 
     ('Sex',LabelBinarizer()), 
     ('Group',LabelBinarizer()), 
     ('familySize',None), 
     ('familyType',LabelBinarizer()), 
     ('Title',LabelBinarizer()) 
    ] 


pipe = Pipeline([ 
    ('featurize', DataFrameMapper(mapping)), 
    ('logReg', LogisticRegression()) 
    ]) 

X = df_train[df_train.columns.drop('Survived')] 
y = df_train['Survived'] 

#model = pipe.fit(X = X, y = y) 
#prediction = model.predict(df_train) 

score = cross_val_score(pipe, X = X, y = y, scoring = 'accuracy') 

df_trainは成果を含め、すべての私のトレーニングセットを含むパンダのデータフレームです。 2つのコメントされた行：

model = pipe.fit(X = X, y = y) 
prediction = model.predict(df_train) 

予期した結果を持つ配列が返されます。

X has 20 features per sample; expecting 19 

完全なコードの下、Kaggle（https://www.kaggle.com/c/titanic/data）にタイタニックCSVファイルを実行することができます

#%% Libraries import 
import pandas as pd 
import numpy as np 
from sklearn_pandas import DataFrameMapper, cross_val_score 

from sklearn.preprocessing import LabelBinarizer 
from sklearn.pipeline import Pipeline 
from sklearn.linear_model import LogisticRegression 

#%% Read the data 

path = 'E:/Kaggle/Titanic/Data/' 
file_training = 'train.csv' 
file_test = 'test.csv' 

#Import the training and test dataset and concatenate them 
df_training = pd.read_csv(path + file_training, header = 0, index_col = 'PassengerId') 
df_test = pd.read_csv(path + file_test, header = 0, index_col = 'PassengerId') 

# Work on the concatenated training and test data for feature engineering and clean-up 
df = pd.concat([df_training, df_test], keys = ['train','test']) 


#%% Initial data exploration and cleaning 

df.describe(include = 'all') 
pd.isnull(df).sum() > 0 

#%% Preprocesing and Cleanup 

#Create new columns with the name (to identify individuals part of a family) 
df['LName'] = df['Name'].apply(lambda x:x.split(',')[0].strip()) 
df['FName'] = df['Name'].apply(lambda x:x.split(',')[1].split('.')[1].strip()) 

#Get the title 
df['Title'] = df['Name'].apply(lambda x:x.split(',')[1].split('.')[0].strip()) 

titleDic = { 
     'Master' : 'kid', 
     'Mlle' : 'unmarriedWoman', 
     'Miss' : 'unmarriedWoman', 
     'Ms' : 'unmarriedWoman', 
     'Jonkheer' : 'noble', 
     'Don' : 'noble', 
     'Dona' : 'noble', 
     'Sir' : 'noble', 
     'Lady' : 'noble', 
     'the Countess' : 'noble', 
     'Capt' : 'ranked', 
     'Major' : 'ranked', 
     'Col' : 'ranked', 
     'Mr' : 'standard', 
     'Mme' : 'standard', 
     'Mrs' : 'standard', 
     'Dr' : 'academic', 
     'Rev' : 'academic' 
     } 
df['Group'] = df['Title'].map(titleDic) 

#%% Working with the family size 

#Get the family size 
df['familySize'] = df['Parch'] + df['SibSp'] + 1 

#Add a family tag (single, couple, small, large) 
df['familyType'] = pd.cut(df['familySize'], 
    [1,2,3,5,np.inf], 
    labels = ['single','couple','sFamily','bFamily'], 
    right = False) 

#%% Filling empty values 
#Fill empty values with the mean or mode for the column 

#Fill the missing values with mean for age per title, class and gender. Store value in AgeFull variable 
agePivot = pd.DataFrame(df.groupby(['Group', 'Sex'])['Age'].median()) 
agePivot.columns = ['AgeFull'] 
df = pd.merge(df, agePivot, left_on = ['Group', 'Sex'], right_index = True) 
df.loc[df['Age'].isnull(),['Age']] = df['AgeFull'] 

#Embark location missing values 
embarkPivot = pd.DataFrame(df.groupby(['Group'])['Embarked'].agg(lambda x:x.value_counts().index[0])) 
embarkPivot.columns = ['embarkFull'] 
df = pd.merge(df, embarkPivot, left_on = ['Group'], right_index = True) 
df.loc[df['Embarked'].isnull(),['Embarked']] = df['embarkFull'] 

#Fill the missing fare value 
df.loc[df['Fare'].isnull(), 'Fare'] = df['Fare'].mean() 

#%% Final clean-up (drop temporary columns) 

df = df.drop(['AgeFull', 'embarkFull'], 1) 

#%% Preparation for training 

df_train = df.loc['train'] 
df_test = df.loc['test'] 

#Creation of dummy variables 

mapping = [ 
      ('Age', None), 
      ('Embarked',LabelBinarizer()), 
      ('Fare',None), 
      ('Pclass',LabelBinarizer()), 
      ('Sex',LabelBinarizer()), 
      ('Group',LabelBinarizer()), 
      ('familySize',None), 
      ('familyType',LabelBinarizer()), 
      ('Title',LabelBinarizer()) 
     ] 

pipe = Pipeline(steps = [ 
     ('featurize', DataFrameMapper(mapping)), 
     ('logReg', LogisticRegression()) 
     ]) 

#Uncommenting the line below fixes the code - why? 
#df_train = df_train.sort_index() 

X = df_train[df_train.columns.drop(['Survived'])] 
y = df_train.Survived 

score = cross_val_score(pipe, X = df_train, y = df_train.Survived, scoring = 'accuracy') 

Answer 1

これは非常に興味深いです：cross_val_scoreと同じを使用して、私は次のエラーを取得します。パイプラインのcross_val_scoreに渡す前にDataFrameのインデックスを使って並べ替えるだけで問題を解決しました。

df_train = df_train.sort_index() 

これがScikitの仕組みに影響する理由は誰にも分かりますか？