Optuna를 사용하여 XGBoost 최적 하이퍼 파라미터 구하는 예제코드

Optuna를 사용하여 XGBoost의  최적 하이퍼 파라미터 구하는 예제코드입니다.

2022. 03. 12  최초작성

# 참고한 코드  https://github.com/optuna/optuna-examples/blob/main/xgboost/xgboost_simple.py import numpy as np import optuna from sklearn import datasets import sklearn.datasets import sklearn.metrics from sklearn.model_selection import train_test_split import xgboost as xgb import psutil import time def memory_usage(message: str = 'debug'):     # current process RAM usage     p = psutil.Process()     rss = p.memory_info().rss / 2 ** 20 # Bytes to MB     print(f"[{message}] memory usage: {rss: 10.5f} MB") iris = datasets.load_iris() data = iris.data target = iris.target # train 데이터세트와 test 데이터세트로 분리 X_train, X_test, y_train, y_test = train_test_split(data, target, test_size=0.33, random_state=1234) # train 데이터세트와 validation 데이터세트로 분리 X_train, X_validation, y_train, y_validation = train_test_split(X_train, y_train, test_size=0.25, random_state=1234) print('X_train.shape, X_test.shape, X_validation.shape', X_train.shape, X_test.shape, X_validation.shape) def objective(trial):     params = {         "objective": "multi:softprob",         "eval_metric":'mlogloss',         "booster": 'gbtree',         #'tree_method':'gpu_hist', 'predictor':'gpu_predictor', 'gpu_id': 0, # GPU 사용시         "tree_method": 'exact', 'gpu_id': -1,  # CPU 사용시         "verbosity": 0,         'num_class':3,         "max_depth": trial.suggest_int("max_depth", 4, 10),         "learning_rate": trial.suggest_uniform('learning_rate', 0.0001, 0.99),         'n_estimators': trial.suggest_int("n_estimators", 1000, 10000, step=100),         "colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),         "colsample_bylevel": trial.suggest_float("colsample_bylevel", 0.5, 1.0),         "colsample_bynode": trial.suggest_float("colsample_bynode", 0.5, 1.0),         "reg_lambda": trial.suggest_loguniform("reg_lambda", 1e-2, 1),         "reg_alpha": trial.suggest_loguniform("reg_alpha", 1e-2, 1),         'subsample': trial.suggest_discrete_uniform('subsample', 0.6, 1.0, 0.05),              'min_child_weight': trial.suggest_int('min_child_weight', 2, 15),         "gamma": trial.suggest_float("gamma", 0.1, 1.0, log=True),         # 'num_parallel_tree': trial.suggest_int("num_parallel_tree", 1, 500) 추가하면 느려짐.     }     model = xgb.XGBClassifier(**params, random_state = 1234, use_label_encoder = False)     bst = model.fit(X_train, y_train,eval_set=[(X_validation,y_validation)], early_stopping_rounds=50, verbose=False)     preds = bst.predict(X_validation)     pred_labels = np.rint(preds)     accuracy = sklearn.metrics.accuracy_score(y_validation, pred_labels)     return accuracy if __name__ == "__main__":     train_start = time.time()     study = optuna.create_study(direction="maximize")     study.optimize(objective, n_trials=50, show_progress_bar=True)     print("Number of finished trials: ", len(study.trials))     print("Best trial:")     trial = study.best_trial     print("  Accuracy: {}".format(trial.value))     print("  Best hyperparameters: ")     for key, value in trial.params.items():         print("    {}: {}".format(key, value))         clf = xgb.XGBClassifier(**study.best_params, random_state = 1234, use_label_encoder = False)     clf.fit(X_train, y_train)     preds = clf.predict(X_test)     # pred_labels = np.rint(preds)     accuracy = sklearn.metrics.accuracy_score(y_test, preds)     print("Accuracy: {}".format(accuracy))     memory_usage("학습하는데 걸린 시간  {:.2f} 분\n".format( (time.time() - train_start)/60))

실행 결과입니다.

 

<frozen importlib._bootstrap>:219: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject

/Users/webnautes/miniforge3/envs/tensorflow-dev/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.

  from pandas import MultiIndex, Int64Index

X_train.shape, X_test.shape, X_validation.shape (75, 4) (50, 4) (25, 4)

[I 2022-03-12 10:36:28,568] A new study created in memory with name: no-name-bedfdb5a-be77-4f99-a78b-485f7b8dd8ef

/Users/webnautes/miniforge3/envs/tensorflow-dev/lib/python3.8/site-packages/optuna/progress_bar.py:47: ExperimentalWarning: Progress bar is experimental (supported from v1.2.0). The interface can change in the future.

  self._init_valid()

[I 2022-03-12 10:36:28,587] Trial 0 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.9140038574107415, 'n_estimators': 8900, 'colsample_bytree': 0.5774527021723073, 'colsample_bylevel': 0.8418824399187912, 'colsample_bynode': 0.8542557602520224, 'reg_lambda': 0.48442484598015073, 'reg_alpha': 0.4154623994036448, 'subsample': 0.9, 'min_child_weight': 2, 'gamma': 0.6976377323928357}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,605] Trial 1 finished with value: 0.96 and parameters: {'max_depth': 7, 'learning_rate': 0.9650561695839202, 'n_estimators': 6000, 'colsample_bytree': 0.9882596223026223, 'colsample_bylevel': 0.8705701751772861, 'colsample_bynode': 0.6591125687103048, 'reg_lambda': 0.035198417054377595, 'reg_alpha': 0.09082510749437141, 'subsample': 0.65, 'min_child_weight': 8, 'gamma': 0.21131337890167107}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,630] Trial 2 finished with value: 0.8 and parameters: {'max_depth': 6, 'learning_rate': 0.6151217932571584, 'n_estimators': 8400, 'colsample_bytree': 0.7511551236163436, 'colsample_bylevel': 0.9824554258648517, 'colsample_bynode': 0.7159129755319047, 'reg_lambda': 0.010683512288834194, 'reg_alpha': 0.7458222808593296, 'subsample': 0.7, 'min_child_weight': 11, 'gamma': 0.1370812836693322}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,646] Trial 3 finished with value: 0.96 and parameters: {'max_depth': 5, 'learning_rate': 0.2637075728899555, 'n_estimators': 8700, 'colsample_bytree': 0.5618576376034451, 'colsample_bylevel': 0.7017043914615777, 'colsample_bynode': 0.7309344654130936, 'reg_lambda': 0.7387081613797699, 'reg_alpha': 0.05652264328828957, 'subsample': 0.8, 'min_child_weight': 3, 'gamma': 0.702298903073803}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,657] Trial 4 finished with value: 0.96 and parameters: {'max_depth': 5, 'learning_rate': 0.2243145397808726, 'n_estimators': 7700, 'colsample_bytree': 0.7568325651122849, 'colsample_bylevel': 0.9401659743604257, 'colsample_bynode': 0.8854312117925491, 'reg_lambda': 0.16488705742049936, 'reg_alpha': 0.07496857309933568, 'subsample': 1.0, 'min_child_weight': 5, 'gamma': 0.15580710938603892}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,672] Trial 5 finished with value: 0.96 and parameters: {'max_depth': 10, 'learning_rate': 0.21404152707049517, 'n_estimators': 6200, 'colsample_bytree': 0.5756217934672239, 'colsample_bylevel': 0.9294552487825101, 'colsample_bynode': 0.9842016712373398, 'reg_lambda': 0.043751571222796194, 'reg_alpha': 0.051303662211252994, 'subsample': 0.65, 'min_child_weight': 3, 'gamma': 0.34039906972082695}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,689] Trial 6 finished with value: 0.36 and parameters: {'max_depth': 8, 'learning_rate': 0.968666430506072, 'n_estimators': 8500, 'colsample_bytree': 0.5586727806328756, 'colsample_bylevel': 0.6944136974430211, 'colsample_bynode': 0.580230395696544, 'reg_lambda': 0.4765008044828287, 'reg_alpha': 0.04967801707856917, 'subsample': 0.75, 'min_child_weight': 15, 'gamma': 0.6052254454022099}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,712] Trial 7 finished with value: 0.96 and parameters: {'max_depth': 9, 'learning_rate': 0.5545861820624186, 'n_estimators': 5600, 'colsample_bytree': 0.5323135654886909, 'colsample_bylevel': 0.9120880282678291, 'colsample_bynode': 0.9767028430942684, 'reg_lambda': 0.061227472491267976, 'reg_alpha': 0.4649840617175656, 'subsample': 0.75, 'min_child_weight': 6, 'gamma': 0.2279063230643731}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,730] Trial 8 finished with value: 0.96 and parameters: {'max_depth': 7, 'learning_rate': 0.145401256272245, 'n_estimators': 2700, 'colsample_bytree': 0.6260498187751011, 'colsample_bylevel': 0.9722894756972016, 'colsample_bynode': 0.5444453972598493, 'reg_lambda': 0.017062707384949586, 'reg_alpha': 0.2794691473108814, 'subsample': 0.75, 'min_child_weight': 6, 'gamma': 0.6007617492345455}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,746] Trial 9 finished with value: 0.36 and parameters: {'max_depth': 4, 'learning_rate': 0.871978017130319, 'n_estimators': 1200, 'colsample_bytree': 0.6468427077740971, 'colsample_bylevel': 0.7149597386172991, 'colsample_bynode': 0.7911136403177917, 'reg_lambda': 0.012844974293704414, 'reg_alpha': 0.07625430516584303, 'subsample': 0.6, 'min_child_weight': 13, 'gamma': 0.7523272136191971}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,765] Trial 10 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.733297754991557, 'n_estimators': 9800, 'colsample_bytree': 0.8751212931557318, 'colsample_bylevel': 0.5603180776956134, 'colsample_bynode': 0.849387539370506, 'reg_lambda': 0.22264598612456238, 'reg_alpha': 0.012937510972652567, 'subsample': 0.95, 'min_child_weight': 2, 'gamma': 0.38392529309087764}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,789] Trial 11 finished with value: 0.96 and parameters: {'max_depth': 7, 'learning_rate': 0.7978109856298516, 'n_estimators': 4100, 'colsample_bytree': 0.9717544469063868, 'colsample_bylevel': 0.8213883396687712, 'colsample_bynode': 0.655427985151957, 'reg_lambda': 0.037143735489469136, 'reg_alpha': 0.24159604294850742, 'subsample': 0.9, 'min_child_weight': 9, 'gamma': 0.21085812996904163}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,814] Trial 12 finished with value: 0.96 and parameters: {'max_depth': 6, 'learning_rate': 0.9770939475761975, 'n_estimators': 6700, 'colsample_bytree': 0.999030811904354, 'colsample_bylevel': 0.8053424294865699, 'colsample_bynode': 0.642030192567808, 'reg_lambda': 0.09251466743127092, 'reg_alpha': 0.16754160429257609, 'subsample': 0.85, 'min_child_weight': 9, 'gamma': 0.1027341008957604}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:28,838] Trial 13 finished with value: 0.96 and parameters: {'max_depth': 8, 'learning_rate': 0.4206111351523977, 'n_estimators': 4500, 'colsample_bytree': 0.8530788136927552, 'colsample_bylevel': 0.8334723371763321, 'colsample_bynode': 0.8380449969628929, 'reg_lambda': 0.3322684570693176, 'reg_alpha': 0.019924921985640373, 'subsample': 0.85, 'min_child_weight': 7, 'gamma': 0.9749979255740981}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,065] Trial 14 finished with value: 0.76 and parameters: {'max_depth': 10, 'learning_rate': 0.013045526070771696, 'n_estimators': 9700, 'colsample_bytree': 0.6822371657010545, 'colsample_bylevel': 0.8618735218878582, 'colsample_bynode': 0.9050227820756218, 'reg_lambda': 0.9643429438905634, 'reg_alpha': 0.932081929253891, 'subsample': 0.6, 'min_child_weight': 11, 'gamma': 0.3998118379946065}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,083] Trial 15 finished with value: 0.96 and parameters: {'max_depth': 5, 'learning_rate': 0.6787493065911958, 'n_estimators': 6900, 'colsample_bytree': 0.8370851864796306, 'colsample_bylevel': 0.7670389068021876, 'colsample_bynode': 0.6596432485881729, 'reg_lambda': 0.023900291743490187, 'reg_alpha': 0.13627303890990125, 'subsample': 0.9, 'min_child_weight': 4, 'gamma': 0.23372296856712382}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,102] Trial 16 finished with value: 0.92 and parameters: {'max_depth': 6, 'learning_rate': 0.8678553906012348, 'n_estimators': 4300, 'colsample_bytree': 0.946309231208169, 'colsample_bylevel': 0.6186380263321656, 'colsample_bynode': 0.7807606041292449, 'reg_lambda': 0.11064755041646827, 'reg_alpha': 0.029998786199702376, 'subsample': 1.0, 'min_child_weight': 8, 'gamma': 0.49265084425798494}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,127] Trial 17 finished with value: 0.96 and parameters: {'max_depth': 8, 'learning_rate': 0.471227399762637, 'n_estimators': 3000, 'colsample_bytree': 0.8194706913141833, 'colsample_bylevel': 0.7707300509575363, 'colsample_bynode': 0.8314845783463175, 'reg_lambda': 0.4132828313185898, 'reg_alpha': 0.011688918849746183, 'subsample': 0.85, 'min_child_weight': 2, 'gamma': 0.9503121536464658}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,150] Trial 18 finished with value: 0.96 and parameters: {'max_depth': 5, 'learning_rate': 0.7063420613837894, 'n_estimators': 7200, 'colsample_bytree': 0.7972640893881355, 'colsample_bylevel': 0.6433392256739143, 'colsample_bynode': 0.5993500141932973, 'reg_lambda': 0.024849825855056976, 'reg_alpha': 0.14205891698261447, 'subsample': 0.95, 'min_child_weight': 4, 'gamma': 0.2667239452639883}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,168] Trial 19 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.7242924875216539, 'n_estimators': 7900, 'colsample_bytree': 0.8883649191580615, 'colsample_bylevel': 0.7861194010138911, 'colsample_bynode': 0.7102703381677948, 'reg_lambda': 0.0728021642732418, 'reg_alpha': 0.4581122030983382, 'subsample': 0.9, 'min_child_weight': 4, 'gamma': 0.28238461635631423}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,191] Trial 20 finished with value: 0.96 and parameters: {'max_depth': 8, 'learning_rate': 0.4129315709898883, 'n_estimators': 2500, 'colsample_bytree': 0.6923344257725652, 'colsample_bylevel': 0.7464688505696225, 'colsample_bynode': 0.9238540200714058, 'reg_lambda': 0.43424166473018716, 'reg_alpha': 0.011388252172018913, 'subsample': 0.85, 'min_child_weight': 2, 'gamma': 0.941514730592551}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,220] Trial 21 finished with value: 0.96 and parameters: {'max_depth': 5, 'learning_rate': 0.44582218344480273, 'n_estimators': 7100, 'colsample_bytree': 0.7978794068482941, 'colsample_bylevel': 0.6273548553674831, 'colsample_bynode': 0.5009091675881632, 'reg_lambda': 0.25702768025895095, 'reg_alpha': 0.3099520496908551, 'subsample': 0.95, 'min_child_weight': 2, 'gamma': 0.7695230814507497}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,241] Trial 22 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.7486629672545314, 'n_estimators': 7800, 'colsample_bytree': 0.9050837631523669, 'colsample_bylevel': 0.6527244735456597, 'colsample_bynode': 0.5927839368340917, 'reg_lambda': 0.0770572217473764, 'reg_alpha': 0.4611935067659485, 'subsample': 0.95, 'min_child_weight': 4, 'gamma': 0.2703706266243239}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,265] Trial 23 finished with value: 0.96 and parameters: {'max_depth': 9, 'learning_rate': 0.3480118959637572, 'n_estimators': 1100, 'colsample_bytree': 0.6970674147036904, 'colsample_bylevel': 0.7491695745503804, 'colsample_bynode': 0.918209995615353, 'reg_lambda': 0.1300764044421201, 'reg_alpha': 0.49774295611098485, 'subsample': 0.9, 'min_child_weight': 3, 'gamma': 0.5203300466320858}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,291] Trial 24 finished with value: 0.96 and parameters: {'max_depth': 6, 'learning_rate': 0.3832344795798284, 'n_estimators': 9100, 'colsample_bytree': 0.7048973297978376, 'colsample_bylevel': 0.5748340282964041, 'colsample_bynode': 0.9434605763574997, 'reg_lambda': 0.22353290733349848, 'reg_alpha': 0.285761517810471, 'subsample': 0.8, 'min_child_weight': 2, 'gamma': 0.8204333396211194}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,327] Trial 25 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.5693353515677614, 'n_estimators': 7700, 'colsample_bytree': 0.9336908028014409, 'colsample_bylevel': 0.5160506433204912, 'colsample_bynode': 0.5057246092161266, 'reg_lambda': 0.25935432083509885, 'reg_alpha': 0.5746847764229592, 'subsample': 0.95, 'min_child_weight': 5, 'gamma': 0.4428998269242421}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,347] Trial 26 finished with value: 0.96 and parameters: {'max_depth': 9, 'learning_rate': 0.8337606467094646, 'n_estimators': 1000, 'colsample_bytree': 0.6221809267958263, 'colsample_bylevel': 0.6549046716212863, 'colsample_bynode': 0.7780915270640204, 'reg_lambda': 0.14389194148841747, 'reg_alpha': 0.44236684568812534, 'subsample': 1.0, 'min_child_weight': 5, 'gamma': 0.5377433480986544}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,372] Trial 27 finished with value: 0.96 and parameters: {'max_depth': 9, 'learning_rate': 0.34653895977081584, 'n_estimators': 10000, 'colsample_bytree': 0.7085899299181204, 'colsample_bylevel': 0.5718999084269273, 'colsample_bynode': 0.941316726098351, 'reg_lambda': 0.5938786204504074, 'reg_alpha': 0.21418090919571198, 'subsample': 0.8, 'min_child_weight': 3, 'gamma': 0.6657561447699961}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,399] Trial 28 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.5832596348873478, 'n_estimators': 9100, 'colsample_bytree': 0.6088240514098432, 'colsample_bylevel': 0.5047208625314985, 'colsample_bynode': 0.8754371557335238, 'reg_lambda': 0.22851935678482255, 'reg_alpha': 0.7309330240175846, 'subsample': 0.8, 'min_child_weight': 6, 'gamma': 0.4470412726749874}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,418] Trial 29 finished with value: 0.96 and parameters: {'max_depth': 7, 'learning_rate': 0.8868636058230864, 'n_estimators': 5200, 'colsample_bytree': 0.5267436958036686, 'colsample_bylevel': 0.876984828380197, 'colsample_bynode': 0.7726838592840303, 'reg_lambda': 0.3047535474463092, 'reg_alpha': 0.6426852282640335, 'subsample': 1.0, 'min_child_weight': 5, 'gamma': 0.5341384058073492}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,440] Trial 30 finished with value: 0.96 and parameters: {'max_depth': 9, 'learning_rate': 0.7965679199840772, 'n_estimators': 9800, 'colsample_bytree': 0.6554194309665206, 'colsample_bylevel': 0.5809909621084746, 'colsample_bynode': 0.8226769205702119, 'reg_lambda': 0.6248763292684465, 'reg_alpha': 0.18297907113469442, 'subsample': 0.7, 'min_child_weight': 7, 'gamma': 0.6336741342016539}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,468] Trial 31 finished with value: 0.96 and parameters: {'max_depth': 10, 'learning_rate': 0.6268986429571514, 'n_estimators': 9000, 'colsample_bytree': 0.6019416145578641, 'colsample_bylevel': 0.5046716330281115, 'colsample_bynode': 0.8808944162258193, 'reg_lambda': 0.16816629282708495, 'reg_alpha': 0.36046741265197374, 'subsample': 0.8, 'min_child_weight': 6, 'gamma': 0.43790717134200285}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,494] Trial 32 finished with value: 0.96 and parameters: {'max_depth': 6, 'learning_rate': 0.9221964876317317, 'n_estimators': 5000, 'colsample_bytree': 0.5016741024356749, 'colsample_bylevel': 0.8753140394644663, 'colsample_bynode': 0.8657063357373233, 'reg_lambda': 0.7086730027860287, 'reg_alpha': 0.6626083147538745, 'subsample': 0.7, 'min_child_weight': 7, 'gamma': 0.6371059315529064}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,520] Trial 33 finished with value: 0.88 and parameters: {'max_depth': 5, 'learning_rate': 0.7924370203974866, 'n_estimators': 9400, 'colsample_bytree': 0.5076070556764618, 'colsample_bylevel': 0.5289691951370902, 'colsample_bynode': 0.7528517630148914, 'reg_lambda': 0.34407201685234223, 'reg_alpha': 0.9895372347698522, 'subsample': 0.7, 'min_child_weight': 10, 'gamma': 0.48065034183244365}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,545] Trial 34 finished with value: 0.96 and parameters: {'max_depth': 6, 'learning_rate': 0.515921120118167, 'n_estimators': 8200, 'colsample_bytree': 0.732545341176866, 'colsample_bylevel': 0.5997145841834561, 'colsample_bynode': 0.9450159678260104, 'reg_lambda': 0.1364921748633317, 'reg_alpha': 0.5622002108186698, 'subsample': 0.9, 'min_child_weight': 3, 'gamma': 0.854593902161743}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,584] Trial 35 finished with value: 0.96 and parameters: {'max_depth': 10, 'learning_rate': 0.6317169266928327, 'n_estimators': 9100, 'colsample_bytree': 0.5839352562994236, 'colsample_bylevel': 0.884981196629837, 'colsample_bynode': 0.8124518612723007, 'reg_lambda': 0.9840665582929615, 'reg_alpha': 0.3352585702373397, 'subsample': 0.7, 'min_child_weight': 7, 'gamma': 0.6913668991973055}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,604] Trial 36 finished with value: 0.96 and parameters: {'max_depth': 5, 'learning_rate': 0.4599936102238196, 'n_estimators': 6200, 'colsample_bytree': 0.786443270733573, 'colsample_bylevel': 0.7955995709402482, 'colsample_bynode': 0.9998045548961173, 'reg_lambda': 0.45609688170689483, 'reg_alpha': 0.09298896625756234, 'subsample': 0.85, 'min_child_weight': 2, 'gamma': 0.17161820495937882}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,629] Trial 37 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.2993085046207926, 'n_estimators': 7400, 'colsample_bytree': 0.769100283673977, 'colsample_bylevel': 0.6749965858754372, 'colsample_bynode': 0.5054712447681706, 'reg_lambda': 0.07129137842898536, 'reg_alpha': 0.11308709654658287, 'subsample': 0.95, 'min_child_weight': 3, 'gamma': 0.3282720915990633}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,659] Trial 38 finished with value: 0.96 and parameters: {'max_depth': 5, 'learning_rate': 0.1569256393875469, 'n_estimators': 2000, 'colsample_bytree': 0.8880871717132108, 'colsample_bylevel': 0.7294490926366143, 'colsample_bynode': 0.5502250649777574, 'reg_lambda': 0.5362567548399997, 'reg_alpha': 0.02286654290701144, 'subsample': 0.95, 'min_child_weight': 2, 'gamma': 0.8356738709591612}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,683] Trial 39 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.2685034442167936, 'n_estimators': 8400, 'colsample_bytree': 0.743224199990103, 'colsample_bylevel': 0.681419217480425, 'colsample_bynode': 0.6126878631790793, 'reg_lambda': 0.05337834659356904, 'reg_alpha': 0.343633650376859, 'subsample': 0.9, 'min_child_weight': 4, 'gamma': 0.369747689005557}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,711] Trial 40 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.5118142730589311, 'n_estimators': 6600, 'colsample_bytree': 0.9318821045787244, 'colsample_bylevel': 0.6523337215310145, 'colsample_bynode': 0.7010350714852848, 'reg_lambda': 0.10172064761630144, 'reg_alpha': 0.4180141276826082, 'subsample': 0.95, 'min_child_weight': 3, 'gamma': 0.7343181710622111}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,731] Trial 41 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.3648874142188512, 'n_estimators': 8000, 'colsample_bytree': 0.9381357626098679, 'colsample_bylevel': 0.5447802645015959, 'colsample_bynode': 0.9591145945437376, 'reg_lambda': 0.1911744411021617, 'reg_alpha': 0.593385354581754, 'subsample': 0.9, 'min_child_weight': 4, 'gamma': 0.30127877072640913}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,751] Trial 42 finished with value: 0.96 and parameters: {'max_depth': 5, 'learning_rate': 0.5602049099735326, 'n_estimators': 8700, 'colsample_bytree': 0.7212726365232496, 'colsample_bylevel': 0.60171931762669, 'colsample_bynode': 0.9047888151143635, 'reg_lambda': 0.12537687583067125, 'reg_alpha': 0.2542024435662278, 'subsample': 1.0, 'min_child_weight': 3, 'gamma': 0.5614495063010848}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,772] Trial 43 finished with value: 0.96 and parameters: {'max_depth': 9, 'learning_rate': 0.3653502155660223, 'n_estimators': 1400, 'colsample_bytree': 0.6509326368806062, 'colsample_bylevel': 0.5308046550564178, 'colsample_bynode': 0.9091570745393864, 'reg_lambda': 0.1505844467830965, 'reg_alpha': 0.5297221473105214, 'subsample': 1.0, 'min_child_weight': 5, 'gamma': 0.542841083617431}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,791] Trial 44 finished with value: 0.96 and parameters: {'max_depth': 8, 'learning_rate': 0.9325354237871785, 'n_estimators': 3600, 'colsample_bytree': 0.6711117311176766, 'colsample_bylevel': 0.7131047253626595, 'colsample_bynode': 0.8550902429237862, 'reg_lambda': 0.18502804917336793, 'reg_alpha': 0.8661299171403399, 'subsample': 1.0, 'min_child_weight': 5, 'gamma': 0.5800414818081224}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,809] Trial 45 finished with value: 0.32 and parameters: {'max_depth': 9, 'learning_rate': 0.21249846405174128, 'n_estimators': 10000, 'colsample_bytree': 0.710628179675153, 'colsample_bylevel': 0.5719773413310978, 'colsample_bynode': 0.9365599935582608, 'reg_lambda': 0.29571285478772436, 'reg_alpha': 0.23198688250472652, 'subsample': 0.75, 'min_child_weight': 15, 'gamma': 0.6802883885862793}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,838] Trial 46 finished with value: 0.96 and parameters: {'max_depth': 7, 'learning_rate': 0.2991517016081809, 'n_estimators': 9400, 'colsample_bytree': 0.5681112214633031, 'colsample_bylevel': 0.587864278074741, 'colsample_bynode': 0.9705517469596477, 'reg_lambda': 0.5896850845400412, 'reg_alpha': 0.20016639360808214, 'subsample': 0.8, 'min_child_weight': 5, 'gamma': 0.8137283620702122}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,865] Trial 47 finished with value: 0.96 and parameters: {'max_depth': 4, 'learning_rate': 0.5759674132832742, 'n_estimators': 8800, 'colsample_bytree': 0.6183325266552224, 'colsample_bylevel': 0.5037791596410508, 'colsample_bynode': 0.7989366213314322, 'reg_lambda': 0.3575738746339738, 'reg_alpha': 0.7591958130983538, 'subsample': 0.75, 'min_child_weight': 6, 'gamma': 0.4293552992851811}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,893] Trial 48 finished with value: 0.96 and parameters: {'max_depth': 9, 'learning_rate': 0.6637571744676964, 'n_estimators': 5800, 'colsample_bytree': 0.613075798655371, 'colsample_bylevel': 0.5596751625979015, 'colsample_bynode': 0.8865441992690419, 'reg_lambda': 0.7599495082682197, 'reg_alpha': 0.7513908186068464, 'subsample': 0.75, 'min_child_weight': 6, 'gamma': 0.4519643712799485}. Best is trial 0 with value: 0.96.

[I 2022-03-12 10:36:29,931] Trial 49 finished with value: 0.96 and parameters: {'max_depth': 10, 'learning_rate': 0.8424798493562657, 'n_estimators': 9300, 'colsample_bytree': 0.5498852953537061, 'colsample_bylevel': 0.9980204941325823, 'colsample_bynode': 0.6772045670904553, 'reg_lambda': 0.255856887930902, 'reg_alpha': 0.4005010000459615, 'subsample': 0.8, 'min_child_weight': 8, 'gamma': 0.3593446245294698}. Best is trial 0 with value: 0.96.

100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:01<00:00, 36.85it/s]

Number of finished trials:  50

Best trial:

  Accuracy: 0.96

  Best hyperparameters: 

    max_depth: 4

    learning_rate: 0.9140038574107415

    n_estimators: 8900

    colsample_bytree: 0.5774527021723073

    colsample_bylevel: 0.8418824399187912

    colsample_bynode: 0.8542557602520224

    reg_lambda: 0.48442484598015073

    reg_alpha: 0.4154623994036448

    subsample: 0.9

    min_child_weight: 2

    gamma: 0.6976377323928357

Accuracy: 0.98

[학습하는데 걸린 시간  0.04 분

] memory usage:  405.37500 MB