Python List의 append와 Numpy 배열의 append 비교

Python에서 List에 원소를 추가하는데 걸리는 시간과 Numpy에서 넘파이 배열에 원소를 추가하는데 걸리는 시간을 비교해봤습니다. 

예상과 달리 넘파이 배열에 원소를 추가하는 시간이 더 오래 걸립니다. 

2021. 11. 29 - 최초작성

Python List에 원소를 추가한 후, 넘파이 배열로 변환하는 방법과 빈 넘파이 배열에 원소를 추가하는 방법 두가지에 대한 코드와 결과입니다. 

 

Python의 List에 원소 추가

 

import numpy as np import time start = time.time() arr = [] for i in range(1000000):     arr.append(i) arr = np.array(arr) print(arr.shape) print("list append time :", time.time() - start)

1000000의 정수를 추가하는데 0.11초 걸렸습니다.

 

(1000000,)

list append time : 0.11541223526000977

 

Numpy의 배열에 원소 추가

 

import numpy as np import time start = time.time() arr = np.array([]) for i in range(1000000):     arr = np.append(arr, i) print(arr.shape) print("numpy append time :", time.time() - start)

1000000의 정수를 추가하는데 380초 걸렸습니다.

(1000000,)

numpy append time : 380.4620773792267

list append가 numpy append 보다 빠른 이유입니다. ( 출처 https://github.com/numpy/numpy/issues/17090 )

 

The reason that list.append() is "fast" is because list over-allocates memory for the items it contains, roughly 1/8 extra items than there are in the list. While there are still free slots at the end of the over-allocated memory, list.append() just puts the new item into a free slot. It only has to re-allocate memory once those over-allocated slots are used up. Because the size of the over-allocated memory grows with the size of the list list.append() is roughly constant-time, on average, for building a long list.

 

numpy does not over-allocate memory. It represents an N-D array, not just a 1-D list, so it can't really over-allocate in all axes. This isn't a matter of whether append() is a function or a method; the data model for numpy arrays just doesn't mesh with the over-allocation strategy that makes list.append() "fast".

 

There are a variety of strategies to build long 1-D arrays quickly. If you know the target length, you can create the array up-front using np.empty() and then filling it in. If you don't know the target length, accumulating values in a list, then using np.array() at the end to convert it to an array is a good option.

참고 

 

https://getpocket.com/read/3067341635