Optuna를 사용한 Keras 분류 모델 하이퍼파라미터 최적화 - iris

Optuna를 사용하여 iris 데이터셋에 대한 Keras 분류 모델 하이퍼파라미터를 최적화하는 예제입니다.

2023. 10. 20 최초작성

최적화 시도를150번 한 결과입니다.

[I 2023-10-20 15:34:32,846] A new study created in memory with name: no-name-a9a9d87a-1258-4aee-91ac-09818702060f Metal device set to: Apple M1 systemMemory: 16.00 GB maxCacheSize: 5.33 GB [I 2023-10-20 15:34:38,095] Trial 0 finished with value: 0.2703423798084259 and parameters: {'n_units_l1': 55, 'dropout': 0.7621006400431453, 'lr': 0.0032892038705258143, 'batch_size': 37}. Best is trial 0 with value: 0.2703423798084259. [I 2023-10-20 15:34:42,519] Trial 1 finished with value: 1.3212648630142212 and parameters: {'n_units_l1': 20, 'dropout': 0.7820854016175295, 'lr': 5.372021881700968e-06, 'batch_size': 58}. Best is trial 0 with value: 0.2703423798084259. [I 2023-10-20 15:34:48,356] Trial 2 finished with value: 0.710401713848114 and parameters: {'n_units_l1': 22, 'dropout': 0.6339466387546254, 'lr': 0.0009254004351235644, 'batch_size': 46}. Best is trial 0 with value: 0.2703423798084259. [I 2023-10-20 15:34:52,139] Trial 3 finished with value: 0.7015336155891418 and parameters: {'n_units_l1': 47, 'dropout': 0.2727602741790787, 'lr': 0.0010879512673758673, 'batch_size': 104}. Best is trial 0 with value: 0.2703423798084259. [I 2023-10-20 15:34:55,586] Trial 4 finished with value: 1.1021714210510254 and parameters: {'n_units_l1': 4, 'dropout': 0.3620519707052726, 'lr': 1.0017643610629562e-05, 'batch_size': 120}. Best is trial 0 with value: 0.2703423798084259. [I 2023-10-20 15:35:00,301] Trial 5 finished with value: 1.1061550378799438 and parameters: {'n_units_l1': 47, 'dropout': 0.23016290900438785, 'lr': 2.3083398990084775e-06, 'batch_size': 88}. Best is trial 0 with value: 0.2703423798084259. [I 2023-10-20 15:35:06,110] Trial 6 finished with value: 0.27406251430511475 and parameters: {'n_units_l1': 37, 'dropout': 0.5827540174961783, 'lr': 0.06605028866717444, 'batch_size': 16}. Best is trial 0 with value: 0.2703423798084259. [I 2023-10-20 15:35:09,805] Trial 7 finished with value: 1.5531014204025269 and parameters: {'n_units_l1': 8, 'dropout': 0.4212584158682844, 'lr': 1.9507740137517812e-06, 'batch_size': 118}. Best is trial 0 with value: 0.2703423798084259. [I 2023-10-20 15:35:16,916] Trial 8 finished with value: 1.1653212308883667 and parameters: {'n_units_l1': 19, 'dropout': 0.5264268031891856, 'lr': 2.2541281519586815e-06, 'batch_size': 15}. Best is trial 0 with value: 0.2703423798084259. [I 2023-10-20 15:35:23,654] Trial 9 finished with value: 1.253339171409607 and parameters: {'n_units_l1': 38, 'dropout': 0.5968164170928301, 'lr': 1.7062751656361422e-06, 'batch_size': 8}. Best is trial 0 with value: 0.2703423798084259. [I 2023-10-20 15:35:28,741] Trial 10 finished with value: 0.18993403017520905 and parameters: {'n_units_l1': 63, 'dropout': 0.8758183640925404, 'lr': 0.018474780678647877, 'batch_size': 26}. Best is trial 10 with value: 0.18993403017520905. [I 2023-10-20 15:35:33,071] Trial 11 finished with value: 0.19209639728069305 and parameters: {'n_units_l1': 64, 'dropout': 0.8956800910057655, 'lr': 0.02473765076303498, 'batch_size': 24}. Best is trial 10 with value: 0.18993403017520905. [I 2023-10-20 15:35:37,073] Trial 12 finished with value: 0.3011542856693268 and parameters: {'n_units_l1': 64, 'dropout': 0.8843027475199228, 'lr': 0.08699809521751403, 'batch_size': 24}. Best is trial 10 with value: 0.18993403017520905. [I 2023-10-20 15:35:42,242] Trial 13 finished with value: 0.1445329785346985 and parameters: {'n_units_l1': 64, 'dropout': 0.898409481795545, 'lr': 0.011182576299713361, 'batch_size': 26}. Best is trial 13 with value: 0.1445329785346985. [I 2023-10-20 15:35:49,155] Trial 14 finished with value: 0.13436166942119598 and parameters: {'n_units_l1': 54, 'dropout': 0.7502511499626234, 'lr': 0.00801192711754471, 'batch_size': 15}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:35:54,897] Trial 15 finished with value: 0.9104092717170715 and parameters: {'n_units_l1': 52, 'dropout': 0.7213521484500983, 'lr': 7.054201773715577e-05, 'batch_size': 12}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:36:02,358] Trial 16 finished with value: 0.2615812420845032 and parameters: {'n_units_l1': 54, 'dropout': 0.13433286221025276, 'lr': 0.005430722068168042, 'batch_size': 9}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:36:09,320] Trial 17 finished with value: 1.0073891878128052 and parameters: {'n_units_l1': 31, 'dropout': 0.6998919635881793, 'lr': 0.00012522342767120007, 'batch_size': 17}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:36:14,275] Trial 18 finished with value: 0.20627163350582123 and parameters: {'n_units_l1': 44, 'dropout': 0.7870434165906445, 'lr': 0.006599434136395845, 'batch_size': 33}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:36:21,899] Trial 19 finished with value: 0.4444827139377594 and parameters: {'n_units_l1': 53, 'dropout': 0.8281668996343743, 'lr': 0.0008025749015354941, 'batch_size': 11}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:36:28,373] Trial 20 finished with value: 0.44796761870384216 and parameters: {'n_units_l1': 58, 'dropout': 0.6739966522198523, 'lr': 0.016978523476022838, 'batch_size': 20}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:36:33,598] Trial 21 finished with value: 0.21152161061763763 and parameters: {'n_units_l1': 59, 'dropout': 0.8450388129775084, 'lr': 0.017626517066434255, 'batch_size': 26}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:36:38,041] Trial 22 finished with value: 0.4150039255619049 and parameters: {'n_units_l1': 64, 'dropout': 0.8886087947439443, 'lr': 0.002435094390098641, 'batch_size': 49}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:36:43,429] Trial 23 finished with value: 0.3083593249320984 and parameters: {'n_units_l1': 59, 'dropout': 0.8238956366003765, 'lr': 0.03430079118669412, 'batch_size': 29}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:36:49,884] Trial 24 finished with value: 0.20651328563690186 and parameters: {'n_units_l1': 42, 'dropout': 0.7372591646212182, 'lr': 0.009062744840565932, 'batch_size': 20}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:36:57,179] Trial 25 finished with value: 0.20894849300384521 and parameters: {'n_units_l1': 60, 'dropout': 0.4741567973771225, 'lr': 0.002067562429397003, 'batch_size': 13}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:37:02,072] Trial 26 finished with value: 0.8406069278717041 and parameters: {'n_units_l1': 49, 'dropout': 0.8104007285520739, 'lr': 0.00025969517504140585, 'batch_size': 36}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:37:06,818] Trial 27 finished with value: 0.21915845572948456 and parameters: {'n_units_l1': 31, 'dropout': 0.662584515268971, 'lr': 0.04107209821018768, 'batch_size': 62}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:37:12,617] Trial 28 finished with value: 0.15880988538265228 and parameters: {'n_units_l1': 58, 'dropout': 0.7480100203720531, 'lr': 0.010518717166137271, 'batch_size': 20}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:37:18,878] Trial 29 finished with value: 0.20388251543045044 and parameters: {'n_units_l1': 53, 'dropout': 0.7606127970729102, 'lr': 0.0036292737904368593, 'batch_size': 20}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:37:23,867] Trial 30 finished with value: 0.24802374839782715 and parameters: {'n_units_l1': 57, 'dropout': 0.5827119324886296, 'lr': 0.008686387391444455, 'batch_size': 43}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:37:29,079] Trial 31 finished with value: 0.22276514768600464 and parameters: {'n_units_l1': 61, 'dropout': 0.8601117656489676, 'lr': 0.013313175341281333, 'batch_size': 27}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:37:35,629] Trial 32 finished with value: 0.29678308963775635 and parameters: {'n_units_l1': 56, 'dropout': 0.7669547134356244, 'lr': 0.04143813315623394, 'batch_size': 21}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:37:43,075] Trial 33 finished with value: 0.34839484095573425 and parameters: {'n_units_l1': 62, 'dropout': 0.8038784336532043, 'lr': 0.0014028470116236962, 'batch_size': 15}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:37:47,774] Trial 34 finished with value: 0.7086274027824402 and parameters: {'n_units_l1': 49, 'dropout': 0.7274154757568746, 'lr': 0.00044081651171465276, 'batch_size': 33}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:37:55,526] Trial 35 finished with value: 0.15267865359783173 and parameters: {'n_units_l1': 56, 'dropout': 0.6365328951560922, 'lr': 0.004792070610758837, 'batch_size': 10}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:38:03,535] Trial 36 finished with value: 0.21673810482025146 and parameters: {'n_units_l1': 51, 'dropout': 0.6272961070671619, 'lr': 0.005190214311722607, 'batch_size': 10}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:38:10,868] Trial 37 finished with value: 0.1884719878435135 and parameters: {'n_units_l1': 43, 'dropout': 0.6732373081951155, 'lr': 0.0031315687864677177, 'batch_size': 14}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:38:17,597] Trial 38 finished with value: 0.7702670097351074 and parameters: {'n_units_l1': 56, 'dropout': 0.4904267862704572, 'lr': 2.7436142861384082e-05, 'batch_size': 17}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:38:25,389] Trial 39 finished with value: 0.39444342255592346 and parameters: {'n_units_l1': 24, 'dropout': 0.5355294335695068, 'lr': 0.0008471887730233741, 'batch_size': 11}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:38:32,087] Trial 40 finished with value: 0.42067578434944153 and parameters: {'n_units_l1': 46, 'dropout': 0.3326586579177233, 'lr': 0.0004205598608639911, 'batch_size': 8}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:38:39,158] Trial 41 finished with value: 0.15085260570049286 and parameters: {'n_units_l1': 41, 'dropout': 0.645114364226875, 'lr': 0.003960738597169333, 'batch_size': 14}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:38:46,322] Trial 42 finished with value: 0.2055334597826004 and parameters: {'n_units_l1': 38, 'dropout': 0.626696311607954, 'lr': 0.010261329074720497, 'batch_size': 18}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:38:53,766] Trial 43 finished with value: 0.25462591648101807 and parameters: {'n_units_l1': 34, 'dropout': 0.5432561593119812, 'lr': 0.001602248695279056, 'batch_size': 13}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:39:01,830] Trial 44 finished with value: 0.17823322117328644 and parameters: {'n_units_l1': 50, 'dropout': 0.6062590840131046, 'lr': 0.003618866490707365, 'batch_size': 10}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:39:08,302] Trial 45 finished with value: 0.32391154766082764 and parameters: {'n_units_l1': 55, 'dropout': 0.43731514506462044, 'lr': 0.05768721569188223, 'batch_size': 23}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:39:16,335] Trial 46 finished with value: 0.15489521622657776 and parameters: {'n_units_l1': 41, 'dropout': 0.684526142034667, 'lr': 0.005667892987051821, 'batch_size': 9}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:42:08,558] Trial 47 finished with value: 0.2113787680864334 and parameters: {'n_units_l1': 26, 'dropout': 0.7010311615323167, 'lr': 0.004396456633581759, 'batch_size': 9}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:42:16,806] Trial 48 finished with value: 0.2972617447376251 and parameters: {'n_units_l1': 40, 'dropout': 0.564527600375847, 'lr': 0.02572851302818229, 'batch_size': 9}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:42:24,123] Trial 49 finished with value: 1.1403404474258423 and parameters: {'n_units_l1': 46, 'dropout': 0.6385604530778929, 'lr': 5.582874410288653e-06, 'batch_size': 11}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:42:28,498] Trial 50 finished with value: 0.39189112186431885 and parameters: {'n_units_l1': 19, 'dropout': 0.6994599007848753, 'lr': 0.005804979413114317, 'batch_size': 81}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:42:36,070] Trial 51 finished with value: 0.26590248942375183 and parameters: {'n_units_l1': 9, 'dropout': 0.7538216147518051, 'lr': 0.01211379980500602, 'batch_size': 14}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:42:42,113] Trial 52 finished with value: 0.17219804227352142 and parameters: {'n_units_l1': 34, 'dropout': 0.7876568533652379, 'lr': 0.02412541176424396, 'batch_size': 16}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:42:47,713] Trial 53 finished with value: 0.15612547099590302 and parameters: {'n_units_l1': 62, 'dropout': 0.6610026505061446, 'lr': 0.007458680697348667, 'batch_size': 12}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:42:54,443] Trial 54 finished with value: 0.22077833116054535 and parameters: {'n_units_l1': 61, 'dropout': 0.658528572338236, 'lr': 0.0026979763341512717, 'batch_size': 8}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:43:01,853] Trial 55 finished with value: 0.14666634798049927 and parameters: {'n_units_l1': 41, 'dropout': 0.7002855788423643, 'lr': 0.006927435110012242, 'batch_size': 10}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:43:10,023] Trial 56 finished with value: 0.3288256824016571 and parameters: {'n_units_l1': 36, 'dropout': 0.7077689294211673, 'lr': 0.0013251820538126677, 'batch_size': 10}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:43:18,370] Trial 57 finished with value: 0.41452279686927795 and parameters: {'n_units_l1': 40, 'dropout': 0.6069580768632376, 'lr': 0.0006933794661965722, 'batch_size': 9}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:43:24,278] Trial 58 finished with value: 0.17881648242473602 and parameters: {'n_units_l1': 45, 'dropout': 0.5638193700447005, 'lr': 0.0021986460869446557, 'batch_size': 12}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:43:31,636] Trial 59 finished with value: 0.24282802641391754 and parameters: {'n_units_l1': 47, 'dropout': 0.8377443976354652, 'lr': 0.01586307930816809, 'batch_size': 13}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:43:38,444] Trial 60 finished with value: 0.664279043674469 and parameters: {'n_units_l1': 41, 'dropout': 0.13676470464209856, 'lr': 0.0247230703416051, 'batch_size': 8}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:43:44,105] Trial 61 finished with value: 0.19017915427684784 and parameters: {'n_units_l1': 63, 'dropout': 0.6807864875527666, 'lr': 0.008538056420928272, 'batch_size': 12}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:43:52,247] Trial 62 finished with value: 0.2642157971858978 and parameters: {'n_units_l1': 31, 'dropout': 0.6411919474841523, 'lr': 0.007598454542699058, 'batch_size': 11}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:44:00,201] Trial 63 finished with value: 0.22664880752563477 and parameters: {'n_units_l1': 54, 'dropout': 0.6878325722008461, 'lr': 0.005823066115927724, 'batch_size': 10}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:44:07,400] Trial 64 finished with value: 0.19696545600891113 and parameters: {'n_units_l1': 60, 'dropout': 0.7256717674183928, 'lr': 0.004074478972555255, 'batch_size': 15}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:44:14,501] Trial 65 finished with value: 0.6491729617118835 and parameters: {'n_units_l1': 48, 'dropout': 0.7767637484041909, 'lr': 0.08620701510241256, 'batch_size': 18}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:45:28,852] Trial 66 finished with value: 0.19005964696407318 and parameters: {'n_units_l1': 64, 'dropout': 0.6481501311278569, 'lr': 0.002004312621538598, 'batch_size': 12}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:45:37,272] Trial 67 finished with value: 0.19955408573150635 and parameters: {'n_units_l1': 52, 'dropout': 0.8609458986237665, 'lr': 0.012466865471689053, 'batch_size': 9}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:45:45,315] Trial 68 finished with value: 0.14769773185253143 and parameters: {'n_units_l1': 36, 'dropout': 0.5822984622741837, 'lr': 0.006899319232319889, 'batch_size': 11}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:45:52,446] Trial 69 finished with value: 0.2899378836154938 and parameters: {'n_units_l1': 36, 'dropout': 0.5118041956087382, 'lr': 0.01861310398794388, 'batch_size': 14}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:45:57,251] Trial 70 finished with value: 0.3014225661754608 and parameters: {'n_units_l1': 43, 'dropout': 0.5887291806727923, 'lr': 0.035519722466496095, 'batch_size': 42}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:46:05,695] Trial 71 finished with value: 0.16580568253993988 and parameters: {'n_units_l1': 39, 'dropout': 0.5585669651149968, 'lr': 0.006875188279618502, 'batch_size': 10}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:46:14,191] Trial 72 finished with value: 0.1444827914237976 and parameters: {'n_units_l1': 58, 'dropout': 0.610644637745982, 'lr': 0.003326834367229645, 'batch_size': 11}. Best is trial 14 with value: 0.13436166942119598. [I 2023-10-20 15:46:21,630] Trial 73 finished with value: 0.12564639747142792 and parameters: {'n_units_l1': 58, 'dropout': 0.6109646682189067, 'lr': 0.004708329408182701, 'batch_size': 11}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:46:29,850] Trial 74 finished with value: 0.19366689026355743 and parameters: {'n_units_l1': 29, 'dropout': 0.62293681723654, 'lr': 0.0027717962184199136, 'batch_size': 11}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:46:37,387] Trial 75 finished with value: 0.5554257035255432 and parameters: {'n_units_l1': 57, 'dropout': 0.45994055819694335, 'lr': 0.00020058314300101985, 'batch_size': 13}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:46:43,380] Trial 76 finished with value: 0.24350722134113312 and parameters: {'n_units_l1': 59, 'dropout': 0.5833268670812177, 'lr': 0.0018537222330674691, 'batch_size': 16}. Best is trial 73 with value: 0.12564639747142792. ^[[B^[[B^[[B[I 2023-10-20 15:46:50,931] Trial 77 finished with value: 0.1999678611755371 and parameters: {'n_units_l1': 55, 'dropout': 0.6122363385670353, 'lr': 0.003741161112054392, 'batch_size': 15}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:46:55,857] Trial 78 finished with value: 0.56975919008255 and parameters: {'n_units_l1': 57, 'dropout': 0.5207006688349215, 'lr': 0.00059416822862648, 'batch_size': 55}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:47:04,091] Trial 79 finished with value: 0.520149290561676 and parameters: {'n_units_l1': 15, 'dropout': 0.5504706414253224, 'lr': 0.0010765647050155812, 'batch_size': 11}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:47:10,824] Trial 80 finished with value: 0.329803466796875 and parameters: {'n_units_l1': 52, 'dropout': 0.5877039968822074, 'lr': 0.015339620518268175, 'batch_size': 18}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:47:19,764] Trial 81 finished with value: 0.17313481867313385 and parameters: {'n_units_l1': 33, 'dropout': 0.7403027832551644, 'lr': 0.004614962686820726, 'batch_size': 10}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:47:27,928] Trial 82 finished with value: 1.2570101022720337 and parameters: {'n_units_l1': 38, 'dropout': 0.7170304501446504, 'lr': 1.046591059616621e-06, 'batch_size': 9}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:47:35,394] Trial 83 finished with value: 0.19310839474201202 and parameters: {'n_units_l1': 42, 'dropout': 0.6738060206979523, 'lr': 0.009935256002050502, 'batch_size': 13}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:47:40,796] Trial 84 finished with value: 0.20484936237335205 and parameters: {'n_units_l1': 44, 'dropout': 0.6478184991519013, 'lr': 0.005730378329387282, 'batch_size': 29}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:47:47,267] Trial 85 finished with value: 0.2986561059951782 and parameters: {'n_units_l1': 36, 'dropout': 0.8039463988669995, 'lr': 0.002941054877096571, 'batch_size': 22}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:47:55,453] Trial 86 finished with value: 0.19564282894134521 and parameters: {'n_units_l1': 59, 'dropout': 0.6885531670155431, 'lr': 0.004496865873011158, 'batch_size': 9}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:48:03,250] Trial 87 finished with value: 0.9427382946014404 and parameters: {'n_units_l1': 50, 'dropout': 0.629868320543828, 'lr': 3.4440704588264404e-05, 'batch_size': 14}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:48:09,782] Trial 88 finished with value: 0.39445701241493225 and parameters: {'n_units_l1': 61, 'dropout': 0.4950072699504069, 'lr': 0.021734891798746273, 'batch_size': 8}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:48:17,552] Trial 89 finished with value: 0.30851835012435913 and parameters: {'n_units_l1': 41, 'dropout': 0.6046403953197815, 'lr': 0.010739199772350869, 'batch_size': 10}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:48:23,335] Trial 90 finished with value: 0.2298663854598999 and parameters: {'n_units_l1': 54, 'dropout': 0.5747802739652136, 'lr': 0.0014352922495187628, 'batch_size': 12}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:48:29,473] Trial 91 finished with value: 0.2938348650932312 and parameters: {'n_units_l1': 61, 'dropout': 0.6607776643481096, 'lr': 0.007759537480447184, 'batch_size': 12}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:48:36,922] Trial 92 finished with value: 0.1413716822862625 and parameters: {'n_units_l1': 58, 'dropout': 0.6591012372898292, 'lr': 0.005843317894396532, 'batch_size': 11}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:48:44,423] Trial 93 finished with value: 0.32258421182632446 and parameters: {'n_units_l1': 58, 'dropout': 0.8974566120047258, 'lr': 0.0032346176404891174, 'batch_size': 11}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:48:52,193] Trial 94 finished with value: 0.20292295515537262 and parameters: {'n_units_l1': 56, 'dropout': 0.7080519279531183, 'lr': 0.005922296322547145, 'batch_size': 11}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:48:59,730] Trial 95 finished with value: 0.3683864176273346 and parameters: {'n_units_l1': 63, 'dropout': 0.6129457677717877, 'lr': 0.014473664007365265, 'batch_size': 13}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:49:08,190] Trial 96 finished with value: 0.17984215915203094 and parameters: {'n_units_l1': 53, 'dropout': 0.3976615951887409, 'lr': 0.0023702622692841476, 'batch_size': 10}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:49:13,952] Trial 97 finished with value: 0.19708538055419922 and parameters: {'n_units_l1': 58, 'dropout': 0.6940388690869956, 'lr': 0.004505141779087402, 'batch_size': 16}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:49:18,882] Trial 98 finished with value: 0.2072726935148239 and parameters: {'n_units_l1': 60, 'dropout': 0.7410690508889877, 'lr': 0.008927706712214193, 'batch_size': 80}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:49:24,328] Trial 99 finished with value: 0.28732895851135254 and parameters: {'n_units_l1': 28, 'dropout': 0.6686825908750085, 'lr': 0.03080087359706803, 'batch_size': 37}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:49:31,139] Trial 100 finished with value: 0.27533552050590515 and parameters: {'n_units_l1': 55, 'dropout': 0.6438609362613411, 'lr': 0.012559733881002497, 'batch_size': 15}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:49:37,120] Trial 101 finished with value: 0.14711487293243408 and parameters: {'n_units_l1': 63, 'dropout': 0.6604804064274438, 'lr': 0.006843344754210686, 'batch_size': 12}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:49:44,302] Trial 102 finished with value: 0.20049864053726196 and parameters: {'n_units_l1': 63, 'dropout': 0.5973418070097661, 'lr': 0.007056376149248901, 'batch_size': 13}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:49:52,329] Trial 103 finished with value: 0.2386772632598877 and parameters: {'n_units_l1': 62, 'dropout': 0.6234244963341373, 'lr': 0.003752006667280409, 'batch_size': 11}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:49:59,406] Trial 104 finished with value: 0.18241913616657257 and parameters: {'n_units_l1': 64, 'dropout': 0.7240116045310461, 'lr': 0.00549537035889697, 'batch_size': 14}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:50:07,580] Trial 105 finished with value: 0.18041813373565674 and parameters: {'n_units_l1': 60, 'dropout': 0.7613943046298227, 'lr': 0.019201616248062176, 'batch_size': 9}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:50:13,431] Trial 106 finished with value: 0.27922943234443665 and parameters: {'n_units_l1': 57, 'dropout': 0.6521939528095115, 'lr': 0.0017643372804592187, 'batch_size': 25}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:50:19,560] Trial 107 finished with value: 0.31228429079055786 and parameters: {'n_units_l1': 39, 'dropout': 0.5359264780953918, 'lr': 0.00891138821194517, 'batch_size': 12}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:50:27,033] Trial 108 finished with value: 0.26686128973960876 and parameters: {'n_units_l1': 37, 'dropout': 0.67204525019359, 'lr': 0.0026266158634151633, 'batch_size': 17}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:50:34,288] Trial 109 finished with value: 0.25249698758125305 and parameters: {'n_units_l1': 32, 'dropout': 0.6850144159604954, 'lr': 0.01127945281489763, 'batch_size': 8}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:50:41,544] Trial 110 finished with value: 0.1459030658006668 and parameters: {'n_units_l1': 51, 'dropout': 0.573158596783322, 'lr': 0.005020054117845072, 'batch_size': 19}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:50:49,430] Trial 111 finished with value: 0.19380851089954376 and parameters: {'n_units_l1': 51, 'dropout': 0.5652867034270914, 'lr': 0.005300497981986012, 'batch_size': 21}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:51:02,697] Trial 112 finished with value: 0.1363895982503891 and parameters: {'n_units_l1': 48, 'dropout': 0.6348154674629674, 'lr': 0.006852092155212644, 'batch_size': 12}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:51:10,028] Trial 113 finished with value: 0.22180135548114777 and parameters: {'n_units_l1': 49, 'dropout': 0.636637749217859, 'lr': 0.0032910713720012046, 'batch_size': 19}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:51:18,002] Trial 114 finished with value: 0.23436163365840912 and parameters: {'n_units_l1': 56, 'dropout': 0.6158149530538024, 'lr': 0.007357434180370048, 'batch_size': 15}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:51:26,034] Trial 115 finished with value: 0.19984693825244904 and parameters: {'n_units_l1': 47, 'dropout': 0.5914235062961356, 'lr': 0.0041981133027439894, 'batch_size': 14}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:51:34,239] Trial 116 finished with value: 0.23827238380908966 and parameters: {'n_units_l1': 54, 'dropout': 0.5749767212887886, 'lr': 0.0067441569448426835, 'batch_size': 17}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:51:42,461] Trial 117 finished with value: 0.21415682137012482 and parameters: {'n_units_l1': 52, 'dropout': 0.8711898111374444, 'lr': 0.01366077076972477, 'batch_size': 13}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:51:48,240] Trial 118 finished with value: 0.20214314758777618 and parameters: {'n_units_l1': 45, 'dropout': 0.630076864611229, 'lr': 0.009473105189199003, 'batch_size': 12}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:51:56,633] Trial 119 finished with value: 0.20016556978225708 and parameters: {'n_units_l1': 53, 'dropout': 0.5999800239695111, 'lr': 0.002190063295112722, 'batch_size': 10}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:52:01,880] Trial 120 finished with value: 0.18813456594944 and parameters: {'n_units_l1': 50, 'dropout': 0.5452146110535355, 'lr': 0.004676911382014975, 'batch_size': 28}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:52:10,838] Trial 121 finished with value: 0.26982197165489197 and parameters: {'n_units_l1': 43, 'dropout': 0.6585857986971374, 'lr': 0.006532113472699982, 'batch_size': 11}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:52:18,020] Trial 122 finished with value: 0.24599510431289673 and parameters: {'n_units_l1': 41, 'dropout': 0.7051542710328728, 'lr': 0.003791605645992706, 'batch_size': 23}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:52:23,450] Trial 123 finished with value: 0.17905545234680176 and parameters: {'n_units_l1': 58, 'dropout': 0.6449851335562468, 'lr': 0.005225312914329731, 'batch_size': 33}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:52:31,834] Trial 124 finished with value: 0.18533794581890106 and parameters: {'n_units_l1': 48, 'dropout': 0.6823829059747049, 'lr': 0.002990121751463613, 'batch_size': 10}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:52:37,788] Trial 125 finished with value: 0.2936908006668091 and parameters: {'n_units_l1': 62, 'dropout': 0.6137196801020911, 'lr': 0.010699565018593265, 'batch_size': 16}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:52:43,790] Trial 126 finished with value: 0.3428248167037964 and parameters: {'n_units_l1': 56, 'dropout': 0.6317286827322569, 'lr': 0.016155041412978813, 'batch_size': 12}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:52:52,744] Trial 127 finished with value: 0.21393878757953644 and parameters: {'n_units_l1': 45, 'dropout': 0.514352484487574, 'lr': 0.008144215104247222, 'batch_size': 9}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:53:05,841] Trial 128 finished with value: 0.23085229098796844 and parameters: {'n_units_l1': 34, 'dropout': 0.3047856227523003, 'lr': 0.006612118866223654, 'batch_size': 11}. Best is trial 73 with value: 0.12564639747142792. [I 2023-10-20 15:53:13,643] Trial 129 finished with value: 0.12363944202661514 and parameters: {'n_units_l1': 59, 'dropout': 0.7131528910388882, 'lr': 0.005214686246654116, 'batch_size': 14}. Best is trial 129 with value: 0.12363944202661514. [I 2023-10-20 15:53:21,192] Trial 130 finished with value: 0.16252483427524567 and parameters: {'n_units_l1': 59, 'dropout': 0.8193094486580078, 'lr': 0.0036987207632484635, 'batch_size': 14}. Best is trial 129 with value: 0.12363944202661514. [I 2023-10-20 15:53:27,248] Trial 131 finished with value: 0.15402238070964813 and parameters: {'n_units_l1': 59, 'dropout': 0.6671396043941631, 'lr': 0.005020867210860354, 'batch_size': 12}. Best is trial 129 with value: 0.12363944202661514. [I 2023-10-20 15:53:33,984] Trial 132 finished with value: 0.19455905258655548 and parameters: {'n_units_l1': 60, 'dropout': 0.7159643768395955, 'lr': 0.004820431716566493, 'batch_size': 12}. Best is trial 129 with value: 0.12363944202661514. [I 2023-10-20 15:53:49,888] Trial 133 finished with value: 0.1661466807126999 and parameters: {'n_units_l1': 55, 'dropout': 0.7804630992416521, 'lr': 0.008788778154400939, 'batch_size': 13}. Best is trial 129 with value: 0.12363944202661514. [I 2023-10-20 15:54:57,235] Trial 134 finished with value: 0.17864000797271729 and parameters: {'n_units_l1': 57, 'dropout': 0.660498396866692, 'lr': 0.002521982956834823, 'batch_size': 14}. Best is trial 129 with value: 0.12363944202661514. [I 2023-10-20 15:55:05,708] Trial 135 finished with value: 0.28883662819862366 and parameters: {'n_units_l1': 58, 'dropout': 0.6749516261599858, 'lr': 0.012555466725223974, 'batch_size': 13}. Best is trial 129 with value: 0.12363944202661514. [I 2023-10-20 15:55:19,820] Trial 136 finished with value: 0.11304563283920288 and parameters: {'n_units_l1': 61, 'dropout': 0.6966387272121367, 'lr': 0.0062223828252492884, 'batch_size': 11}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:55:32,995] Trial 137 finished with value: 0.16176895797252655 and parameters: {'n_units_l1': 62, 'dropout': 0.729102581147305, 'lr': 0.00693988273210961, 'batch_size': 11}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:56:35,013] Trial 138 finished with value: 0.26306748390197754 and parameters: {'n_units_l1': 63, 'dropout': 0.5794211488259475, 'lr': 0.010459777188933967, 'batch_size': 10}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:56:40,828] Trial 139 finished with value: 0.22148281335830688 and parameters: {'n_units_l1': 60, 'dropout': 0.7011632552220035, 'lr': 0.003507680717116097, 'batch_size': 30}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:56:49,052] Trial 140 finished with value: 0.19703857600688934 and parameters: {'n_units_l1': 64, 'dropout': 0.740267148410777, 'lr': 0.017707713969408918, 'batch_size': 19}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:56:58,015] Trial 141 finished with value: 0.2212059050798416 and parameters: {'n_units_l1': 61, 'dropout': 0.6552895464801232, 'lr': 0.005113375258790601, 'batch_size': 15}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:57:04,434] Trial 142 finished with value: 0.15053017437458038 and parameters: {'n_units_l1': 59, 'dropout': 0.6937858329526596, 'lr': 0.005570314555879132, 'batch_size': 12}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:57:22,010] Trial 143 finished with value: 0.2761686146259308 and parameters: {'n_units_l1': 57, 'dropout': 0.18751046422362044, 'lr': 0.006378470531538163, 'batch_size': 11}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:57:28,640] Trial 144 finished with value: 0.21036802232265472 and parameters: {'n_units_l1': 55, 'dropout': 0.6944117099137969, 'lr': 0.008846084064473746, 'batch_size': 12}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:58:31,886] Trial 145 finished with value: 0.9343011975288391 and parameters: {'n_units_l1': 61, 'dropout': 0.7594645328578129, 'lr': 9.801235222085556e-05, 'batch_size': 14}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:58:50,407] Trial 146 finished with value: 0.19484670460224152 and parameters: {'n_units_l1': 59, 'dropout': 0.6408303540684723, 'lr': 0.00286857465353308, 'batch_size': 11}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 15:59:53,412] Trial 147 finished with value: 0.29506781697273254 and parameters: {'n_units_l1': 53, 'dropout': 0.7140928178177599, 'lr': 0.00396215791984529, 'batch_size': 10}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 16:00:15,534] Trial 148 finished with value: 0.30022865533828735 and parameters: {'n_units_l1': 51, 'dropout': 0.6027387929473236, 'lr': 0.0011117222085974966, 'batch_size': 13}. Best is trial 136 with value: 0.11304563283920288. [I 2023-10-20 16:00:22,492] Trial 149 finished with value: 0.3239777982234955 and parameters: {'n_units_l1': 56, 'dropout': 0.6832900417822771, 'lr': 0.007721552147521994, 'batch_size': 12}. Best is trial 136 with value: 0.11304563283920288. Number of finished trials:  150 Best trial: Value:  0.11304563283920288 Params:      n_units_l1: 61     dropout: 0.6966387272121367     lr: 0.0062223828252492884     batch_size: 11

위 실행결과의 마지막에 다음처럼 최적 파라미터가 출력되도록 했습니다.

Number of finished trials:  150 Best trial: Value:  0.11304563283920288 Params:      n_units_l1: 61     dropout: 0.6966387272121367     lr: 0.0062223828252492884     batch_size: 11

학습과정에 대한 Loss와 Accuracy 그래프입니다.

Test 데이터셋으로 성능 평가한 결과입니다.

전체 코드입니다.

 

import numpy as np import optuna from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.metrics import classification_report, confusion_matrix from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Dropout from tensorflow.keras.optimizers import Adam import os from sklearn.preprocessing import StandardScaler import matplotlib.pyplot as plt # 텐서플로우의 INFO와 WARNING 메시지 숨기기 os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' # IRIS 데이터를 불러옵니다. data = load_iris() X = data.data y = data.target # 데이터 분할: Train 80%, Test 20% X_train_full, X_test, y_train_full, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Train 데이터셋을 분할 :  Train 80%, Test 20% Train 75%, 검증 25%가 됨. X_train, X_val, y_train, y_val = train_test_split(X_train_full, y_train_full, test_size=0.2, random_state=42) # 최종 Train : Validation : Test 비율은 64 : 16 : 20입니다 # 학습에 도움이 되도록 데이터 스케일링 scaler = StandardScaler() X_train = scaler.fit_transform(X_train) X_val = scaler.transform(X_val) X_test = scaler.transform(X_test) # 모델을 생성하는 함수입니다. def create_model(trial):     model = Sequential()     # 첫 번째 히든 레이어     model.add(Dense(trial.suggest_int("n_units_l1", 4, 64),  # 뉴런 수: 4~64 사이의 값                     'relu'))         model.add(Dropout(trial.suggest_float("dropout", 0.1, 0.9))) # dropout : 0.1 ~ 0.9 사이의 값         # 출력 레이어     model.add(Dense(3, activation='softmax'))  # 3개의 클래스로 분류하기 때문에 뉴런 수는 3     # 학습률 선택: 1e-5 ~ 1e-1 사이의 값     lr = trial.suggest_float("lr", 1e-6, 1e-1, log=True)         model.compile(loss="sparse_categorical_crossentropy",  # 손실 함수                   optimizer=Adam(learning_rate=lr),  # Adam 옵티마이저                   metrics=["accuracy"])  # 평가 지표     return model def objective(trial):     # 모델 생성     model = create_model(trial)         # 배치 크기 선택: 8 ~ 128 사이의 값     batch_size = trial.suggest_int("batch_size", 8, 128, log=True)         # 모델 훈련     model.fit(X_train, y_train,               validation_data=(X_val, y_val),  # 검증 데이터               epochs=50,  # 에포크 수               batch_size=batch_size,  # 배치 크기               verbose=0)         # 손실값 반환     loss, _ = model.evaluate(X_val, y_val, verbose=0)     return loss # Optuna 학습 객체 생성 study = optuna.create_study(direction="minimize")  # 손실을 최소화하는 방향으로 최적화 study.optimize(objective, n_trials=150)  # 최적화 시도 횟수: 150 print() print('Number of finished trials: ', len(study.trials)) print('Best trial:') trial = study.best_trial print('Value: ', trial.value) print('Params: ') for key, value in trial.params.items():     print(f'    {key}: {value}') print() # 최적의 파라미터로 모델 생성 final_model = create_model(study.best_trial) # 최적의 배치 크기로 훈련 best_batch_size = study.best_trial.params["batch_size"] history = final_model.fit(X_train_full, y_train_full,                           validation_data=(X_val, y_val),                           epochs=50,                           batch_size=best_batch_size,                           verbose=0) # 학습 과정 그래프 plt.figure(figsize=(12, 4)) plt.subplot(1, 2, 1) plt.plot(history.history["loss"], label="Training Loss") plt.plot(history.history["val_loss"], label="Validation Loss") plt.title("Loss") plt.legend() plt.subplot(1, 2, 2) plt.plot(history.history["accuracy"], label="Training Accuracy") plt.plot(history.history["val_accuracy"], label="Validation Accuracy") plt.title("Accuracy") plt.legend() plt.tight_layout() plt.show() # 테스트 데이터로 성능 평가 test_predictions = np.argmax(final_model.predict(X_test), axis=1) print("Classification Report:") print(classification_report(y_test, test_predictions)) print("\nConfusion Matrix:") print(confusion_matrix(y_test, test_predictions))