merve04 Posted September 6, 2020 Share #1 Posted September 6, 2020 I'm not sure if this is where I need to post this but here we go, Yesterday i was tinkering around in docker downloading images, removing old images, doing some backups etc all very simultaneously. Then a box popped up mentioning errors occurred in the file systems.... I've tried doing a simple reboot of the machine but issues still there, and i tried doing the "reboot now" and run a check, maybe 30min went by, maybe less, just recall seeing "checking quota" and then i was able to log in dsm again. Heres some screenshots. Quote Link to comment Share on other sites More sharing options...
flyride Posted September 6, 2020 Share #2 Posted September 6, 2020 Is there a question that needs an answer here? It looks like you must be using ext4 filesystem instead of btrfs and the system decided it had some corruption and recommended an automatic fsck. It can only do that when the ext4 system is unmounted, so that means on boot. Not unusual for it to take 30 minutes or longer, I have some very large blob ext4 filesystems at work that take over 24 hours to fsck. If it were btrfs you would get a similar notification of the error but it would have been fixed on the spot if it was possible via redundancy. Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 6, 2020 Author Share #3 Posted September 6, 2020 Indeed there’s a question, twice I’ve tried reboot now and run, both times come back to dsm stating there’s an error. Is there anything else i can do? Quote Link to comment Share on other sites More sharing options...
flyride Posted September 7, 2020 Share #4 Posted September 7, 2020 The message seems to be saying that DSM cannot fix the filesystem and that you should offload your files, (delete the filesystem), and recreate it. Quote Link to comment Share on other sites More sharing options...
jensmander Posted September 7, 2020 Share #5 Posted September 7, 2020 Synology’s official procedure: https://www.synology.com/en-us/knowledgebase/DSM/tutorial/Storage/What_should_I_do_if_a_file_system_error_occurs As @flyride said, you should backup your files. After that you should run an extended SMART test to see if the error is hardware related. Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 7, 2020 Author Share #6 Posted September 7, 2020 I’ve done a bit of searching on fsck, I guess there are more complex options for scanning the volume?!? Not too sure if that’s a route I should take. My biggest concern is if I indeed need to recreate the volume, how does one approach offloading nearly 40TB?!? Quote Link to comment Share on other sites More sharing options...
flyride Posted September 7, 2020 Share #7 Posted September 7, 2020 How do you backup your data now? I agree that's a lot of data. When it starts to get that voluminous, your backup/restore strategy needs to be adjusted to remain viable and realistic. For me, I keep another "small" XPE box online with very high density drives and its only job is to receive btrfs snapshot replicas. Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 7, 2020 Author Share #8 Posted September 7, 2020 (edited) Being that a 8TB drive costs $200, even if i wanted to be cheap\risky\crazy and degrade my volume to use couple of the redundant drives as means of storage, id be set back $600. I've been looking at cloud options, g suite as an example. I'm also thinking of trying and removing all docker packages and docker app itself and see if that may be the root cause of this error as it did occur while manipulating things in there. Edited September 7, 2020 by merve04 Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 9, 2020 Author Share #9 Posted September 9, 2020 I'm running data scrub on storage pool 1, could it possibly fix the error or am i wasting my time? Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 9, 2020 Author Share #10 Posted September 9, 2020 Well data scrub completed, no errors found but didnt fix the problem 😪 Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 9, 2020 Author Share #11 Posted September 9, 2020 Well I tried my second plan as this all started when I was in docker, I removed docker and selected the check mark to remove all settings, files and folder, while it was removing the message about file system error occurred popped up. When it finished removing i did a run/reboot, this time it took almost 1 hour checking quota, when i logged back in dsm, voila warning was gone. 😎 Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 9, 2020 Author Share #12 Posted September 9, 2020 Well this is annoying, as soon as I reinstalled docker, error came back!!! Quote Link to comment Share on other sites More sharing options...
flyride Posted September 9, 2020 Share #13 Posted September 9, 2020 Can you confirm your filesystem is on ext4? AFAIK Docker is not supported unless you are running on btrfs. Could be part of your problem. # cat /etc/fstab Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 10, 2020 Author Share #14 Posted September 10, 2020 I can confirm my volume is ext4, I can also ensure that I’ve been using docker for probably 2 years now. I do have a second volume as you can see in the picture above for surveillance and the only reason it’s btrfs was to support virtual machine manager. My system is back in checking quota but sure I could install docker on that volume. I’m just lead to suspect something is still lingering after uninstalling docker from volume1 Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 10, 2020 Author Share #15 Posted September 10, 2020 so back in dsm now, error gone, decided to ssh in server, theres still a "@docker" folder with some stuff in it, could this be the cause? safe to delete? Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 10, 2020 Author Share #16 Posted September 10, 2020 so interestingly as i kept digging deeper and deeper in the folder structure of @docker, i had dsm logged in the background and on that last CD command, the system error occurred popped up. Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 11, 2020 Author Share #17 Posted September 11, 2020 Anyone know how I can delete the directory? When I do a rm -rf it’s gives me an error that structure needs cleaning. No matter how many times fsck is ran, the problem remains. Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 12, 2020 Author Share #18 Posted September 12, 2020 So im trying to see if debugfs can remove this folder, but i dont understand why when i try to mount the volume in questions it seems like it just does nothing, i try a "ls" after what I believe should of mounted the volume in debug but nothing happens. Am I missing something, doing something wrong? Quote Link to comment Share on other sites More sharing options...
flyride Posted September 13, 2020 Share #19 Posted September 13, 2020 I don't know much about debugfs but it's really for kernel debugging and passing kernel parameters to userspace. And I am pretty sure you can't use it on a mounted ext4 filesystem. I know you are trying not to offload your files and rebuild with the hope of a quick easy fix. I don't mean to be preachy, but trying to edit your filesystem directly is a path that increasingly puts your data at risk. If fsck can't fix it, there is something significantly wrong. Quote Link to comment Share on other sites More sharing options...
merve04 Posted September 13, 2020 Author Share #20 Posted September 13, 2020 Do you or anyone else seeing this know how to perform e2fsck? Ive also read stuff about it being more effective. 10 hours ago, flyride said: I know you are trying not to offload your files and rebuild with the hope of a quick easy fix. The reason I'm investing my time in this is i can literally access every other folder in my volume no issue, but as soon as i touch @docker, immediately triggers dsm a error and to run a fs. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.