This particular error is generally experienced when people attempt a non NTFS volume such as ReFS to a DFSR replication group as documented on the following forum.
https://social.technet.microsoft.com/Forums/windowsserver/en-US/8860b354-3bf2-43ed-9acf-07fa43931046/dfsr-replication-cant-be-used-on-a-refs-volume?forum=winserver8gen
In my case, I experienced this error with DFSR setup on NTFS volume on a new branch server.
The customer moved the DFSR node from one office, to another office. The C:\ "SYSTEM" volume was formatted and reloaded with a fresh copy of 2012 R2 with latest patches, however the E:\ "DATA" volume was not formatted. As a result it has its legacy DFSR Database, Staging data etc.
To clean up the staging data I created a new directory in C:\ called "empty":
C:\Empty
I then ran the following commands after stopping the DFSR Replication service on the spoke server:
Robocopy "C:\Empty" "E:\System Volume Information\DFSR" /MIR
Followed by
rmdir "E:\System Volume Information\DFSR"
Starting the DFSR replication service resulted in the error above.
After playing around a bit more, I found that in addition to flushing all data in System Volume Information\DFSR you must also remove the DfsrPrivate link which points to the System Volume Information\DFSR sub directory for DfsrPrivate data.
After doing this, starting the DFS Replication service kicks of its normal DFSR Initial Sync shown by a state of 2:
Wmic /namespace:\\root\microsoftdfs path dfsrreplicatedfolderinfo get replicationgroupname,replicatedfoldername,state
State Values (Uninitialized, Initialized, Initial Sync, Auto Recovery, Normal, In Error), ValueMap (0, 1, 2, 3, 4, 5)}
DFSR clean Staging folder
