I2P support forum

MuWire usage and testing

I have been stress testing MuWire versions with both plugin and standalone processes.
I have been liking what I see with the features and ease of use in the current versions and wanted to see how it performed with larger collections of files.

As expected when you start adding over 200,000 files it causes some issues. First it takes several days for MuWire to hash and index all the files anytime you have to reinstall or make a mistake and erase your ID. Also the plugin and the standalone are separate installs so you have to go through a rebuild to change over. Reloading is much quicker than a reinstall so just restarting MuWire only takes an hour or two with huge numbers of files. For someone with only tens of thousands of files it just takes a few minutes.

If you have been trying to download from me and I keep disappearing and coming back with a slightly different ID it is because I am restarting with a different install and making some basic changes to settings. I really appreciate how Zlatin and others have been making a huge effort to make changes to accomodate the usage that I am attempting. Sometimes I get responses with solutions in a few minutes.

Unfortunately, with this many files it often takes 3 days just for MuWire to reinstall. Other times I have to set the upload slots to 0 so that my hard drives do not get beat up too badly. A 3 day reinstall means 3 days of constant thrashing on the drive. I need to stop uploading occasionally so my equipment has a chance to catch up. My apologies to those who have been trying to download from me for the disruptions.

The other thing that may be causing disruptions is that browsing is not enabled on my installs and often subscriptions are disabled as well. These are great features and really useful however with this large of a collection it can crash peoples browsers. In the future testing will happen with that as well but if we are doing too many things at once we start to lose track of where we are and how to change things.

Please keep trying to download and if necessary doing the search for the file again if it disappears. We can only really test it if people are using the application so the next time you want to find something try MuWire instead of Google or DuckDuckGo.

If anyone has questions I will check back here periodically. If you are comfortable with git maybe try helping out with things there.

Thanks for this post mortense. We are very excited to work on making MuWire efficient enough to share large number of files, and while there's been some progress there's still a long way to go until we can confidently say it "just works".

Few quick notes on the points you make:

* The initial hashing indeed can take days - there's nothing that can be done about that. Hashing is an I/O intensive process as every single file that is to be shared needs to be read from beginning to end. Having said that, it is enough to do the hashing just one time, and after the index is created it can be reused by future installs. The index is stored in a directory called "files" under the MuWire home directory, which is either $HOME/.MuWire for standalone or $HOME/.i2p/plugins/MuWire for the plugin. If you do not delete that directory the next re-install will just load the index that is already there. Likewise, if you're switching between plugin and standalone you can symlink that directory in the appropriate home folder and the index will be reused. In fact, one of the recommendations from http://muwire.i2p/many-files.html is to symlink the "files" directory and put it on an SSD mount. That will speed up the loading process a lot.
* The cryptographic identity part of a MuWire ID is stored in a file called "key.dat" in the home directory which can be preserved across installs or shared between plugin and standalone. The only caveat is that you can't run both plugin and standalone at the same time with the same key.dat - I2P won't allow that.
* The human-readable part of the MuWire ID is stored in a file called "MuWire.properties", also in the home directory, under the "nickname" property.
* By default standalone limits the memory heap size to 256MB, and the plugin is limited by the memory available to the router which is also 256MB by default. You want to increase that by at least a factor of 10 if you intend to share 200k+ files. I'll post instructions on how to do that separately.
* Browsing will have to stay disabled since a flat list of files needs to be rendered in a single table, both in plugin and standalone. Unfortunately I don't think any browser is capable of rendering 200k+ rows in a single page, or in a single Swing table for standalone (but happy to be proven wrong). To partially alleviate that I've implemented a new feature called "Collections" which will be released later today where you can group files belonging together in a single collection - like songs from an album for example.

That is all for now.