Improved incremental processing and upload API

Closes #587 (closed). Introduces an additional path_filter argument to the process_upload that enables limiting the processing to a specific path. This is used to optimize the API calls to put and delete individual files and folders at the specified raw path.