Hyperopt提供了一個(gè)優(yōu)化接口，這個(gè)接口接受一個(gè)評(píng)估函數(shù)和參數(shù)空間，能計(jì)算出參數(shù)空間內(nèi)的一個(gè)點(diǎn)的損失函數(shù)值。用戶還要指定空間內(nèi)參數(shù)的分布情況。 Hyheropt四個(gè)重要的因素：指定需要最小化的函數(shù)，搜索的空間，采樣的數(shù)據(jù)集(trails database)（可選），搜索的算法（可選）。 首先，定義一個(gè)目標(biāo)函數(shù),接受一個(gè)變量,計(jì)算后返回一個(gè)函數(shù)的損失值，比如要最小化函數(shù)q(x,y) = x**2 + y**2:
from hyperopt import hpspace = [hp.uniform(’x’, 0, 1), hp.normal(’y’, 0, 1)]然后，定義一個(gè)參數(shù)空間，比如x在0-1區(qū)間內(nèi)取值，y是實(shí)數(shù)，所以
第三，指定搜索的算法，算法也就是hyperopt的fmin函數(shù)的algo參數(shù)的取值。當(dāng)前支持的算法由隨機(jī)搜索(對(duì)應(yīng)是hyperopt.rand.suggest)，模擬退火(對(duì)應(yīng)是hyperopt.anneal.suggest)，TPE算法。舉個(gè)栗子：
from hyperopt import hp, fmin, rand, tpe, space_evalbest = fmin(q, space, algo=rand.suggest)PRint space_eval(space, best)搜索算法本身也有內(nèi)置的參數(shù)決定如何去優(yōu)化目標(biāo)函數(shù)，我們可以指定搜索算法的參數(shù)，比如針對(duì)TPE，指定jobs：
from functools import partialfrom hyperopt import hp, fmin, tpealgo = partial(tpe.suggest, n_startup_jobs=10)best = fmin(q, space, algo=algo)print space_eval(space, best)關(guān)于參數(shù)空間的設(shè)置，比如優(yōu)化函數(shù)q，輸入fmin(q,space=hp.uniform(‘a(chǎn)’,0,1)).hp.uniform函數(shù)的第一個(gè)參數(shù)是標(biāo)簽，每個(gè)超參數(shù)在參數(shù)空間內(nèi)必須具有獨(dú)一無(wú)二的標(biāo)簽。hp.uniform指定了參數(shù)的分布。其他的參數(shù)分布比如 hp.choice返回一個(gè)選項(xiàng)，選項(xiàng)可以是list或者tuple.options可以是嵌套的表達(dá)式，用于組成條件參數(shù)。 hp.pchoice(label,p_options)以一定的概率返回一個(gè)p_options的一個(gè)選項(xiàng)。這個(gè)選項(xiàng)使得函數(shù)在搜索過(guò)程中對(duì)每個(gè)選項(xiàng)的可能性不均勻。 hp.uniform(label,low,high)參數(shù)在low和high之間均勻分布。 hp.quniform(label,low,high,q),參數(shù)的取值是round(uniform(low,high)/q)*q，適用于那些離散的取值。 hp.loguniform(label,low,high)繪制exp(uniform(low,high)),變量的取值范圍是[exp(low),exp(high)] hp.randint(label,upper) 返回一個(gè)在[0,upper)前閉后開(kāi)的區(qū)間內(nèi)的隨機(jī)整數(shù)。 搜索空間可以含有l(wèi)ist和dictionary.
from hyperopt import hplist_space = [hp.uniform(’a’, 0, 1),hp.loguniform(’b’, 0, 1)]tuple_space = (hp.uniform(’a’, 0, 1),hp.loguniform(’b’, 0, 1))dict_space = {’a’: hp.uniform(’a’, 0, 1),’b’: hp.loguniform(’b’, 0, 1)}使用sample函數(shù)從參數(shù)空間內(nèi)采樣：
from hyperopt.pyll.stochasti import sampleprint sample(list_space)# => [0.13, .235]print sample(nested_space)# => [[{’case’: 1, ’a’, 0.12‘}, {’case’: 2, ’b’: 2.3}],# ’extra_literal_string’,# 3]在參數(shù)空間內(nèi)使用函數(shù)：
from hyperopt.pyll import scopedef foo(x):return str(x) ? 3expr_space = {’a’: 1 + hp.uniform(’a’, 0, 1),’b’: scope.minimum(hp.loguniform(’b’, 0, 1), 10),’c’: scope.call(foo, args=(hp.randint(’c’, 5),)),}—————–這是一條有點(diǎn)短的昏割線———————————–
在blog上發(fā)現(xiàn)了一段使用感知器判別鳶尾花數(shù)據(jù)的代碼，使用的學(xué)習(xí)率是0.1,迭代40次得到了一個(gè)測(cè)試集上正確率為82%的結(jié)果。使用hyperopt優(yōu)化參數(shù)，將正確率提升到了91%。
from sklearn import datasetsimport numpy as npfrom sklearn.cross_validation import train_test_splitfrom sklearn.metrics import accuracy_scoreiris = datasets.load_iris()X = iris.datay = iris.targetX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)from sklearn.preprocessing import StandardScalersc = StandardScaler()sc.fit(X_train)X_train_std = sc.transform(X_train)X_test_std = sc.transform(X_test)from sklearn.linear_model import Perceptronppn = Perceptron(n_iter=40, eta0=0.1, random_state=0)ppn.fit(X_train_std, y_train)y_pred = ppn.predict(X_test_std)print accuracy_score(y_test, y_pred)def percept(args): global X_train_std,y_train,y_test ppn = Perceptron(n_iter=int(args["n_iter"]),eta0=args["eta"]*0.01,random_state=0) ppn.fit(X_train_std, y_train) y_pred = ppn.predict(X_test_std) return -accuracy_score(y_test, y_pred)from hyperopt import fmin,tpe,hp,partialspace = {"n_iter":hp.choice("n_iter",range(30,50)), "eta":hp.uniform("eta",0.05,0.5)}algo = partial(tpe.suggest,n_startup_jobs=10)best = fmin(percept,space,algo = algo,max_evals=100)print bestprint percept(best)#0.822222222222#{'n_iter': 14, 'eta': 0.12877033763511717}#-0.911111111111xgboost具有很多的參數(shù)，把xgboost的代碼寫成一個(gè)函數(shù)，然后傳入fmin中進(jìn)行參數(shù)優(yōu)化，將交叉驗(yàn)證的auc作為優(yōu)化目標(biāo)。auc越大越好，由于fmin是求最小值，因此求-auc的最小值。所用的數(shù)據(jù)集是202列的數(shù)據(jù)集，第一列樣本id，最后一列是label,中間200列是屬性。
#coding:utf-8import numpy as npimport pandas as pdfrom sklearn.preprocessing import MinMaxScalerimport xgboost as xgbfrom random import shufflefrom xgboost.sklearn import XGBClassifierfrom sklearn.cross_validation import cross_val_scoreimport pickleimport timefrom hyperopt import fmin, tpe, hp,space_eval,rand,Trials,partial,STATUS_OKdef loadFile(fileName = "E://zalei//browsetop200Pca.csv"): data = pd.read_csv(fileName,header=None) data = data.values return datadata = loadFile()label = data[:,-1]attrs = data[:,:-1]labels = label.reshape((1,-1))label = labels.tolist()[0]minmaxscaler = MinMaxScaler()attrs = minmaxscaler.fit_transform(attrs)index = range(0,len(label))shuffle(index)trainIndex = index[:int(len(label)*0.7)]print len(trainIndex)testIndex = index[int(len(label)*0.7):]print len(testIndex)attr_train = attrs[trainIndex,:]print attr_train.shapeattr_test = attrs[testIndex,:]print attr_test.shapelabel_train = labels[:,trainIndex].tolist()[0]print len(label_train)label_test = labels[:,testIndex].tolist()[0]print len(label_test)print np.mat(label_train).reshape((-1,1)).shapedef GBM(argsDict): max_depth = argsDict["max_depth"] + 5 n_estimators = argsDict['n_estimators'] * 5 + 50 learning_rate = argsDict["learning_rate"] * 0.02 + 0.05 subsample = argsDict["subsample"] * 0.1 + 0.7 min_child_weight = argsDict["min_child_weight"]+1 print "max_depth:" + str(max_depth) print "n_estimator:" + str(n_estimators) print "learning_rate:" + str(learning_rate) print "subsample:" + str(subsample) print "min_child_weight:" + str(min_child_weight) global attr_train,label_train gbm = xgb.XGBClassifier(nthread=4, #進(jìn)程數(shù) max_depth=max_depth, #最大深度 n_estimators=n_estimators, #樹(shù)的數(shù)量 learning_rate=learning_rate, #學(xué)習(xí)率 subsample=subsample, #采樣數(shù) min_child_weight=min_child_weight, #孩子數(shù) max_delta_step = 10, #10步不降則停止 objective="binary:logistic") metric = cross_val_score(gbm,attr_train,label_train,cv=5,scoring="roc_auc").mean() print metric return -metricspace = {"max_depth":hp.randint("max_depth",15), "n_estimators":hp.randint("n_estimators",10), #[0,1,2,3,4,5] -> [50,] "learning_rate":hp.randint("learning_rate",6), #[0,1,2,3,4,5] -> 0.05,0.06 "subsample":hp.randint("subsample",4),#[0,1,2,3] -> [0.7,0.8,0.9,1.0] "min_child_weight":hp.randint("min_child_weight",5), # }algo = partial(tpe.suggest,n_startup_jobs=1)best = fmin(GBM,space,algo=algo,max_evals=4)print bestprint GBM(best)hyperopt文獻(xiàn)原文介紹: 鏈接：http://pan.baidu.com/s/1i5aAXKx 密碼：2gtr
新聞熱點(diǎn)
疑難解答
網(wǎng)友關(guān)注
