參加kaggle數(shù)據(jù)挖掘比賽，就第一個(gè)賽題Titanic的數(shù)據(jù)，學(xué)習(xí)相關(guān)數(shù)據(jù)預(yù)處理以及模型建立，本博客關(guān)注基于pandas進(jìn)行數(shù)據(jù)預(yù)處理過程。包括數(shù)據(jù)統(tǒng)計(jì)、數(shù)據(jù)離散化、數(shù)據(jù)關(guān)聯(lián)性分析

引入包和加載數(shù)據(jù)

import pandas as pdimport numpy as nptrain_df =pd.read_csv('../datas/train.csv') # train settest_df = pd.read_csv('../datas/test.csv') # test setcombine = [train_df, test_df]

清洗數(shù)據(jù)

查看數(shù)據(jù)維度以及類型

缺失值處理

查看object數(shù)據(jù)統(tǒng)計(jì)信息

數(shù)值屬性離散化

計(jì)算特征與target屬性之間關(guān)系

查看數(shù)據(jù)維度以及類型

#查看前五條數(shù)據(jù)print train_df.head(5) #查看每列數(shù)據(jù)類型以及nan情況print train_df.info() # 獲得所有object屬性print train_data.describe(include=['O']).columns 

查看object數(shù)據(jù)統(tǒng)計(jì)信息

#查看連續(xù)數(shù)值屬性基本統(tǒng)計(jì)情況print train_df.describe() #查看object屬性數(shù)據(jù)統(tǒng)計(jì)情況print train_df.describe(include=['O']) # 統(tǒng)計(jì)Title單列各個(gè)元素對應(yīng)的個(gè)數(shù)print train_df['Title'].value_counts() # 屬性列刪除train_df = train_df.drop(['Name', 'PassengerId'], axis=1) 

缺失值處理

# 直接丟棄缺失數(shù)據(jù)列的行print df4.dropna(axis=0,subset=['col1']) # 丟棄nan的行,subset指定查看哪幾列 print df4.dropna(axis=1) # 丟棄nan的列# 采用其他值填充dataset['Cabin'] = dataset['Cabin'].fillna('U') dataset['Title'] = dataset['Title'].fillna(0) # 采用出現(xiàn)最頻繁的值填充freq_port = train_df.Embarked.dropna().mode()[0]dataset['Embarked'] = dataset['Embarked'].fillna(freq_port)# 采用中位數(shù)或者平均數(shù)填充test_df['Fare'].fillna(test_df['Fare'].dropna().median(), inplace=True)test_df['Fare'].fillna(test_df['Fare'].dropna().mean(), inplace=True)

數(shù)值屬性離散化，object屬性數(shù)值化

# 創(chuàng)造一個(gè)新列，F(xiàn)areBand，將連續(xù)屬性Fare切分成四份train_df['FareBand'] = pd.qcut(train_df['Fare'], 4)# 查看切分后的屬性與target屬性Survive的關(guān)系train_df[['FareBand', 'Survived']].groupby(['FareBand'], as_index=False).mean().sort_values(by='FareBand', ascending=True)# 建立object屬性映射字典 title_mapping = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Royalty":5, "Officer": 6}dataset['Title'] = dataset['Title'].map(title_mapping)

計(jì)算特征與target屬性之間關(guān)系

object與連續(xù)target屬性之間，可以groupby均值

object與離散target屬性之間，先將target數(shù)值化，然后groupby均值，或者分別條形統(tǒng)計(jì)圖

連續(xù)屬性需要先切割然后再進(jìn)行g(shù)roupby計(jì)算，或者pearson相關(guān)系數(shù)

print train_df[['AgeBand', 'Survived']].groupby(['AgeBand'], as_index=False).mean().sort_values(by='AgeBand', ascending=True)

總結(jié)pandas基本操作