I built this while dealing with large SMB shares over WiFi, where traditional
directory scanning tools were either too slow or caused unnecessary network load.
The main idea is a layered scanning strategy:
- Minimize SMB round-trips
- Use Rayon for controlled parallelism
- Avoid spawning excessive IO-bound tasks
- Keep memory usage predictable
This is not meant to be a fancy UI tool, but a fast and deterministic building
block that can be embedded into other systems.
The project is open-source (Apache 2.0).
I'm especially interested in feedback on:
- The concurrency model
- SMB traversal strategy
- Any edge cases you've seen in similar systems
Hi HN,
I'm the author of DeepSearch.
I built this while dealing with large SMB shares over WiFi, where traditional directory scanning tools were either too slow or caused unnecessary network load.
The main idea is a layered scanning strategy: - Minimize SMB round-trips - Use Rayon for controlled parallelism - Avoid spawning excessive IO-bound tasks - Keep memory usage predictable
This is not meant to be a fancy UI tool, but a fast and deterministic building block that can be embedded into other systems.
The project is open-source (Apache 2.0). I'm especially interested in feedback on: - The concurrency model - SMB traversal strategy - Any edge cases you've seen in similar systems
Thanks for taking a look.
Hi thx for sharing, in general
about concurrency Rayon excels on CPU-heavy tasks, but this is more IO
for traversal avoid any extra round-trips, simple the metadata already given
edge-cases would be infinite loops on symlinks, the Windows 255 char limits and on WIFI packet loss due to too many reqs