8

This is actually a follow up question of this question. I managed to get the profiling to work and the problem really seems to be lazy evaluation.

The data structure I'm using is a Map Int (Map Int Text), where Text is from Data.Text. The problem is, that the function which builds this map creates a huge thunk. Working on an input text of about 3 MB the programs needs more than 250 MB of memory.

Now to the real purpose of this question:

To get the amount of characters in this data structure is use the following function:

type TextResource = M.Map Int (M.Map Int T.Text)

totalSize :: TextResouce -> Int
totalSize = M.fold ((+) . (M.fold ((+). T.length) 0)) 0

Not beautiful, but it gets the job done. I'm using this function in the main function right after the TextResource is created. The interesting thing is, that when I profile the program by using the RTS option -hr or -hc the memory usage goes down to 70 or 50 MB after a while, which would be totally fine.

Unfortunately this only works when using the profiling options and the totalSize function - without them it's back to 250 MB.

I uploaded the program (< 70 lines) together with a test file and a cabal file, so that you can try it yourself: Link

The test.xml is a generated XML file, which should be put into the executables directory. To build, cabal configure --enable-executable-profiling and afterwards cabal build should be enough (if you have the profiling versions of the required libraries installed).

You can see the change when running the program once with +RTS -hc and once without.

I'd be really great if someone could run the program, since I'm really stuck here. I already tried to put in deepseq at several places, but nothing works (well, besides using the profiling options).

Edit:

Profiling does show, however, that only ~20MB of the heap is used, so as in my comment, I blame GHC for not freeing as much of the GC nursery memory as you seem to want.

Thanks, that pointed me into the correct direction. As it turns out, you can tell GHC to perform a garbage collection (performGC), which works perfectly well, after deepseqing the map. Even though I guess the usage of performGC is not recommended, it seems to be the right tool for the job here.

Edit2: This is how I changed the main function (+ deepseqing the return of buildTextFile):

main = do tf <- buildTextFile "test.xml"
          performGC
          putStrLn . show . text 1 1000 $ tf
          getLine
          putStrLn . show . text 100 1000 $ tf
          return ()
4

1 に答える 1

4

問題は、このマップを構築する関数が巨大なサンクを作成することです。

いいえ。ヒーププロファイリングに基づいて、スペースの使用がサンクであるとは思いません。また、Data.Map厳密な HashMaps に置き換えて、(大きなサンクの作成を避けるために) マップを強制しても同じ結果になりました。

RTS オプション -hr または -hc を使用してプログラムのプロファイルを作成すると、しばらくするとメモリ使用量が 70 MB または 50 MB に低下します。

これを再現できません。、-hr、-hyまたは-hcの場合、プロセスは 140MB のヒープを保持します。ただし、プロファイリングでは、ヒープが最大 20MB しか使用されていないことが示されているため、私のコメントにあるように、GHC が GC ナーサリ メモリを必要なだけ解放していないことを非難します。

-hy プロファイル

計算中の高いメモリ使用量に関しては、上記の-hyプロファイルは、ほとんどのメモリがStringタイプと HaXML ライブラリPosnタイプによるものであることを示しています。よりリソースを意識した XML ライブラリ (xml-enumerator?)を探すという私の提案を繰り返しますByteString。Text

于 2011-07-29T22:41:22.037 に答える