oh shit people have already done it

https://dmitryulyanov.github.io/audio-texture-synthesis-and-style-transfer/